Suspend a thread's pooler credentials on archive instead of deleting them

[albrand]

Human comments

What was wrong

When a thread was archived, the pool deleted its scoped and nested credentials (removeThread). Unarchiving it (restoreThread) only cleared the archive marker. Any session still holding the old tokens then failed every model call with 401 Invalid Account Pooler bearer token. That includes native CLIs, which cannot take a new token, managed turns, and fleet self-archive/unarchive. #193 tried to restore deleted credentials from a tombstone kept for 24 hours. After nine review rounds it still documented an accepted trade-off: a crash partway through an archive meant the thread's credentials were not restored. The owner rejected that trade-off. This PR replaces #193 and removes the failure path instead of narrowing it.

What changed

plugins/account-pool/src/thread-tokens.ts:

• suspendThread (archive) bumps the archive version, marks the thread archived in memory and writes the archive marker (archived: true) at the thread's current epoch, falling back to the fallback marker as before. Records stay loaded and on disk (digests only). Authentication rejects them while the thread is archived, and minting for an archived thread still throws.
• If neither marker can be written, each of the thread's credential files is renamed to suspended-<name> with its content unchanged. Load treats a suspended file as an archived thread's credential: it is loaded and held as archived. If a rename fails too, the file stays where it is and the thread stays suspended in memory.
• restoreThread (unarchive) lifts the suspension at the same epoch, renames suspended files back and reloads anything not yet in memory, so the same tokens are accepted again. It rereads an unreadable marker first; while the marker stays unknown it reloads nothing from disk, so a stale file a delete left behind cannot come back. Lifting the suspension in memory does not depend on the marker write or the renames. A failed one leaves the disk saying "archived", which the next check against bb corrects.
• removeThread is used only for delete (and for a sweep lookup that reports gone). It always advances the epoch and removes the thread's files, including suspended ones.
• bb is asked on every authentication (admit, askBb). authenticate and authenticateNested look up the thread's state with bb on each call. Concurrent calls for the same thread share one in-flight lookup, bounded by the sweep lookup timeout, and no answer is cached beyond that flight. live accepts (and lifts a local suspension), archived rejects and suspends, gone rejects and deletes. A missed archive or unarchive event therefore takes effect on the next request. When the lookup fails or times out, the token is rejected (401) and nothing is deleted or suspended; a failed lookup is retried once within the same flight (not after a timeout). The store logs a warning, at most once a minute per thread:
• bb is asked before every mint and env export (forThread, forNested, admit). Both call admit, the same single-flight askBb that authentication uses, before they take the store lock, and refuse unless bb answers live. A failed or timed-out lookup refuses ("Cannot mint a credential for a thread bb has not confirmed live.") and changes nothing, the same fail-closed rule as authentication. gone deletes through the existing gone path, then the mint throws "Cannot mint a credential for a deleted thread."; archived suspends, then it throws "Cannot mint a credential for an archived thread.". The local deleted set stays as an extra guard, and bb is the authority. The env export (contributeFor) already asked bb before minting (checkArchived, then settleThread) and returns no pool credential for an archived or gone thread; it does not throw, and that is unchanged. Cost: each env resolution now makes two more of bb's in-process thread lookups (bb.sdk.threads.get, one SQLite read each), one for the thread credential and one for the nested credential; a cached credential is still returned without minting, but only after its lookup.

  Account Pooler rejected a token for thread <threadId>: bb's thread lookup failed, so the thread could not be confirmed live.
  

  (timed out replaces failed on a timeout.) The same tokens work again once bb answers live. verifiedThreads is removed. The lookup cost was not measured; bb.sdk.threads.get runs in the same server and no cache was added.
• Archive state changes only under a current version (restoreThread, admit, isArchived). suspendThread and removeThread bump the thread's archive version and set the suspension synchronously, before any await. restoreThread now does its disk work first (marker write, renaming suspended files back, loading credentials). It checks the version before each step and commits the in-memory restore only after a final synchronous check. If an archive or delete arrived meanwhile, it leaves the thread suspended and returns false; the queued suspend or delete then writes its marker last. admit accepts only if the restore committed and the version is unchanged right before it returns, and authenticate / authenticateNested check the version again just before returning a route. isArchived does not apply a re-read marker, or clear a read failure, if the version moved during the read. The env-export path (revokeIfArchived) treats a restore that did not commit as archived. A successful archived: false marker write no longer clears marker-failure state; the restore's commit does.
• Every committed archive-state change advances a generation counter (commitArchiveState). Every write to archivedThreads, activeArchiveMarkers and archiveEpochs goes through it, so it bumps archiveVersion. The writes are: archive and delete (synchronous, before any await), the epoch those two write, a restore's final commit, an unarchive event that finds the thread already live, a marker read from disk, and a suspended file loaded from disk. A commit by an archive or delete also records that version as the thread's last suspension.
  • An archived or gone lookup answer is applied only if the generation has not moved since the lookup started (suspendThreadIfVersion, removeThreadIfVersion).
  • A live answer, and a request's acceptance, are rejected if any archive or delete committed since the answer's version. Applying a live answer can only lift a suspension, so the only later change it could undo is an archive or delete.
  • A stale answer is dropped, never applied. The request then asks bb once more (settle, at most two lookups) and is answered from that, or rejected if the second answer is stale too.
  • An unarchive event (unarchiveThread) always starts its own lookup rather than joining one already in flight. A lookup joins an in-flight one only if both started at the same generation.
  • revokeIfArchived (env export) and the thread.unarchived handler now go through the same settle path as token checks.
• Deletion is final (removeThread, isDeleted). A delete records the thread in deletedThreadDigests (sha256 of the thread id) in its synchronous prelude, and writes deleted-thread-<digest>.json through the same temp, fsync and rename path as the other markers. Startup reads deletions from the marker file names, before any credential file, and deletes any surviving credential file of a deleted thread.
  • forThread and forNested throw for a deleted thread. restoreThread never commits for one.
  • settle returns null for a deleted thread whatever bb answers, which covers token checks, the unarchive handler and env export. When bb answers live after the delete, it logs: Account Pooler kept thread <threadId> deleted: bb reported it live after it was deleted, and a deleted thread's credentials are never restored or minted again.
  • If no record can be written, the deletion holds in memory and removeThread rethrows. The record is retried on the next event, sweep and start (below).
  • Older builds ignore deleted-thread-* files.
• The deletion authority is bb's own database. bb commits a thread's deletion before the plugin hears of it, and never takes it back:
  • The delete route writes deletedAt synchronously in SQLite (markThreadDeleted) before it emits thread.deleted (apps/server/src/routes/threads/base.ts lines 718-722 on ours). The failed-create path emits first and writes in the same synchronous tick (thread-create.ts lines 537-542). The plugin's first write happens only after an await, so it always follows the commit.
  • No server code clears deletedAt. The only writes set it to a time: packages/db/src/data/threads.ts line 2135 and the failed-create emit above.
  • The plugin's kv row is as durable as deletedAt. bb.storage.kv.set runs a synchronous SQLite insert into bb.db, with no buffering, in the same database as deletedAt (plugin-api.ts lines 723-737, packages/db/src/data/plugin-storage.ts lines 33-47).
• Invariant:
  • Credentials are never removed before a deletion record is durable.
  • The plugin never contradicts bb's committed thread state.
  • Residual: none. bb commits the delete before the plugin hears of it, so after a committed delete bb answers gone, and every mint, export and authentication asks bb. That holds even when the plugin's kv write and every filesystem record were refused and nothing survived locally.
• Record order (recordDeletion, persistDeletion, holdDeletedFiles, recordPendingDeletion). On thread.deleted, and on a lookup or sweep answer of gone, removeThread tries in this order:
  1. the atomic deleted-thread-<digest>.json;
  2. the archive marker and its fallback, both with deleted: true, when the thread held credentials;
  3. if none of those landed, the marker created in place (open(…, "w")), then the deleted-thread-<digest>.d directory;
  4. if no filesystem record landed, each credential file renamed to deleted-pending-<digest>-<name>; that rename is itself a record;
  5. every time, bb.storage.kv.set("deleted-thread:<digest>", { deletedAt }) in bb's database.
• A record counts only once it is durable (persist, createDeletedRecord, createDeletedDirectory, holdDeletedFiles, syncDirectory). Every filesystem path counts as landed only after its content write (where it has content), the file sync and a sync of the store directory all succeed. Any error on the write, the file sync or the directory sync means the record did not land, and the next path is tried, exactly as after a refused open. No error is swallowed into success. The pending renames are followed by a directory sync too; if it fails, a warning says so, and the deletion stays unrecorded and is retried.
• Why the kv write is last. It was first in round 10. The round 11 probes showed that kv-first leaves a window in the code, and no test can close it: a refused kv write followed by a crash before the first filesystem record leaves no record anywhere. Writing kv last means a kv refusal can only happen after every filesystem path has already landed or failed.
  • Run 38550550168: bb answers live whenever the subset refuses kv. 340 runs per lifetime accepted an old token or minted. 32 of them crashed before the kv write, where nothing was refused, so bb answering live there was wrong; that was Hermes's round 10 objection.
  • Run 38551897166: bb's answer is derived per run, live only if this run's kv write was attempted and refused. The code still writes kv first. 308 runs per lifetime accepted: 12 in the permitted cells, and 296 in that window.
  • Run 38552038836: the same test with kv written last. Every job passed, the table equals the rule, and only the 12 runs in the permitted cells accept.
• Nothing is removed before a durable record exists. Credential files are removed (rm) only after at least one durable record landed. The kv row lands before the first credential removal whenever kv accepts it. A pending file is removed only once a durable record of another kind exists: kv, a marker or the directory. If a rename fails, that file stays untouched and a warning names it. Nothing is removed in that case, even when rm works. removeThread then throws, and the thread stays deleted in the process: both token paths are rejected and both mints throw.
• Retries. An unrecorded deletion is retried by the next thread.deleted event and by every sweep. A thread known only from its pending files is retried at every start and every sweep. Each retry records first, then removes exactly the files the failed attempt left in place.
• Start. initialize loads every deleted-thread: kv row into the in-memory deleted set, together with the deleted-thread-* markers and the pending files, before any credential file is read (see "Partial markers at start" for when a marker counts). Every later record write updates that set.
  • Authentication never reads kv.
  • Only this start-time load fails closed. If it fails, every thread token is rejected and every mint throws (a mint first retries the load) until a reload succeeds.
  • The load is retried on the existing sweep schedule. A reload that succeeds removes the credentials of any loaded thread it marks deleted.
• A marker that cannot be read (anything but ENOENT) blocks restore and mint until it can be read. It does not count as a confirmed deletion, because that would delete an archived thread's credentials on a single read error.
• Partial markers at start (initialize, confirmsDeletedRecord, confirmsRecord, loadArchiveMarker, loadFile).
  • A deleted-thread-<digest>.json found at start counts as a confirmed deletion only if its content parses as { threadDigest, deleted: true } for that digest and syncing the file and then the directory succeeds. A deleted-thread-<digest>.d directory needs the directory sync. An archive marker with deleted: true needs the same file and directory sync. A marker left by a process crash can still be unsynced, so it is synced before anything relies on it.
  • A partial, empty or unreadable deletion marker still puts the thread in the deleted set, so restore and mint stay blocked. It is not a record: the plugin writes the record again at start and on every sweep, and removes nothing (no pending file, no credential file) until a record lands.
  • loadFile removes a deleted thread's credential files only once its deletion is confirmed. Before, a thread known only from its pending files or from an unconfirmed marker lost its remaining credential files at load.
  • A corrupt archive marker without deleted: true keeps blocking restore and mint as before and is never read as a deletion, so a failed archive write never turns into one.
• Computed table. "computes the deletion residual for every subset of refused bb-database, content, rename, rm and directory mutations, with content refused at its open, its write, its file sync or the directory sync, one and two credential files, and every crash point from the first mutation of the delete through the next start, losing every unsynced change at each restart, minting first on a start that has not swept yet, with a lookup stub answering gone after every restart in place of bb's committed thread row, which the server deletion matrix reads for real" (thread-tokens-suspend.test.ts, both lifetimes). This is the store-level model: its ThreadLookup is a stub that answers gone, and the plugin's store is driven directly. The same matrix through bb's real delete route and bb's restarted database is the "Server deletion matrix" below. Its output, verbatim, from CI run 38584387035 at 5efd26e (the process lifetime prints the same rows). No run accepted an old token or minted.

deletion residual (durable); every unsynced change is lost at each restart, the first start mints and authenticates before any sweep, and the lookup stub answers gone in place of bb's committed deletedAt:
1 file | refused: none | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
1 file | refused: kv | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
1 file | refused: content@open | durable record: yes | old token accepted: no | mint succeeds: no | runs: 18
1 file | refused: content@write | durable record: yes | old token accepted: no | mint succeeds: no | runs: 23
1 file | refused: content@sync | durable record: yes | old token accepted: no | mint succeeds: no | runs: 28
1 file | refused: content@dirsync | durable record: yes | old token accepted: no | mint succeeds: no | runs: 38
1 file | refused: kv+content@open | durable record: yes | old token accepted: no | mint succeeds: no | runs: 18
1 file | refused: kv+content@write | durable record: yes | old token accepted: no | mint succeeds: no | runs: 23
1 file | refused: kv+content@sync | durable record: yes | old token accepted: no | mint succeeds: no | runs: 28
1 file | refused: kv+content@dirsync | durable record: no | old token accepted: no | mint succeeds: no | runs: 43
1 file | refused: rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
1 file | refused: kv+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
1 file | refused: content@open+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 18
1 file | refused: content@write+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 23
1 file | refused: content@sync+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 28
1 file | refused: content@dirsync+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 33
1 file | refused: kv+content@open+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 18
1 file | refused: kv+content@write+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 23
1 file | refused: kv+content@sync+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 28
1 file | refused: kv+content@dirsync+rename | durable record: no | old token accepted: no | mint succeeds: no | runs: 38
1 file | refused: rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 42
1 file | refused: kv+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 42
1 file | refused: content@open+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 20
1 file | refused: content@write+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 25
1 file | refused: content@sync+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
1 file | refused: content@dirsync+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 39
1 file | refused: kv+content@open+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 20
1 file | refused: kv+content@write+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 25
1 file | refused: kv+content@sync+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
1 file | refused: kv+content@dirsync+rm | durable record: no | old token accepted: no | mint succeeds: no | runs: 43
1 file | refused: rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
1 file | refused: kv+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
1 file | refused: content@open+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 20
1 file | refused: content@write+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 25
1 file | refused: content@sync+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
1 file | refused: content@dirsync+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
1 file | refused: kv+content@open+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 20
1 file | refused: kv+content@write+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 25
1 file | refused: kv+content@sync+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
1 file | refused: kv+content@dirsync+rename+rm | durable record: no | old token accepted: no | mint succeeds: no | runs: 38
1 file | refused: mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
1 file | refused: kv+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
1 file | refused: content@open+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 18
1 file | refused: content@write+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 23
1 file | refused: content@sync+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 28
1 file | refused: content@dirsync+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 37
1 file | refused: kv+content@open+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 19
1 file | refused: kv+content@write+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 24
1 file | refused: kv+content@sync+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 29
1 file | refused: kv+content@dirsync+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 42
1 file | refused: rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
1 file | refused: kv+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
1 file | refused: content@open+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 17
1 file | refused: content@write+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 22
1 file | refused: content@sync+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 27
1 file | refused: content@dirsync+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
1 file | refused: kv+content@open+rename+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 22
1 file | refused: kv+content@write+rename+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 27
1 file | refused: kv+content@sync+rename+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 32
1 file | refused: kv+content@dirsync+rename+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 37
1 file | refused: rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 42
1 file | refused: kv+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 42
1 file | refused: content@open+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 19
1 file | refused: content@write+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 24
1 file | refused: content@sync+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 29
1 file | refused: content@dirsync+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 38
1 file | refused: kv+content@open+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 19
1 file | refused: kv+content@write+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 24
1 file | refused: kv+content@sync+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 29
1 file | refused: kv+content@dirsync+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 42
1 file | refused: rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
1 file | refused: kv+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
1 file | refused: content@open+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 18
1 file | refused: content@write+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 23
1 file | refused: content@sync+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 28
1 file | refused: content@dirsync+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 33
1 file | refused: kv+content@open+rename+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 22
1 file | refused: kv+content@write+rename+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 27
1 file | refused: kv+content@sync+rename+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 32
1 file | refused: kv+content@dirsync+rename+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 37
2 files | refused: none | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
2 files | refused: kv | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
2 files | refused: content@open | durable record: yes | old token accepted: no | mint succeeds: no | runs: 20
2 files | refused: content@write | durable record: yes | old token accepted: no | mint succeeds: no | runs: 25
2 files | refused: content@sync | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
2 files | refused: content@dirsync | durable record: yes | old token accepted: no | mint succeeds: no | runs: 41
2 files | refused: kv+content@open | durable record: yes | old token accepted: no | mint succeeds: no | runs: 20
2 files | refused: kv+content@write | durable record: yes | old token accepted: no | mint succeeds: no | runs: 25
2 files | refused: kv+content@sync | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
2 files | refused: kv+content@dirsync | durable record: no | old token accepted: no | mint succeeds: no | runs: 45
2 files | refused: rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: kv+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: content@open+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 20
2 files | refused: content@write+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 25
2 files | refused: content@sync+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
2 files | refused: content@dirsync+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 36
2 files | refused: kv+content@open+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 20
2 files | refused: kv+content@write+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 25
2 files | refused: kv+content@sync+rename | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
2 files | refused: kv+content@dirsync+rename | durable record: no | old token accepted: no | mint succeeds: no | runs: 40
2 files | refused: rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 46
2 files | refused: kv+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 46
2 files | refused: content@open+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 24
2 files | refused: content@write+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 29
2 files | refused: content@sync+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: content@dirsync+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 43
2 files | refused: kv+content@open+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 24
2 files | refused: kv+content@write+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 29
2 files | refused: kv+content@sync+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: kv+content@dirsync+rm | durable record: no | old token accepted: no | mint succeeds: no | runs: 45
2 files | refused: rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 38
2 files | refused: kv+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 38
2 files | refused: content@open+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 24
2 files | refused: content@write+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 29
2 files | refused: content@sync+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: content@dirsync+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 38
2 files | refused: kv+content@open+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 24
2 files | refused: kv+content@write+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 29
2 files | refused: kv+content@sync+rename+rm | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: kv+content@dirsync+rename+rm | durable record: no | old token accepted: no | mint succeeds: no | runs: 40
2 files | refused: mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
2 files | refused: kv+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
2 files | refused: content@open+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 21
2 files | refused: content@write+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 26
2 files | refused: content@sync+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 31
2 files | refused: content@dirsync+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 40
2 files | refused: kv+content@open+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 21
2 files | refused: kv+content@write+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 26
2 files | refused: kv+content@sync+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 31
2 files | refused: kv+content@dirsync+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 44
2 files | refused: rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: kv+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: content@open+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 20
2 files | refused: content@write+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 25
2 files | refused: content@sync+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 30
2 files | refused: content@dirsync+rename+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 35
2 files | refused: kv+content@open+rename+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 24
2 files | refused: kv+content@write+rename+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 29
2 files | refused: kv+content@sync+rename+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: kv+content@dirsync+rename+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 39
2 files | refused: rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 46
2 files | refused: kv+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 46
2 files | refused: content@open+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 23
2 files | refused: content@write+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 28
2 files | refused: content@sync+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 33
2 files | refused: content@dirsync+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 42
2 files | refused: kv+content@open+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 21
2 files | refused: kv+content@write+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 26
2 files | refused: kv+content@sync+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 31
2 files | refused: kv+content@dirsync+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 44
2 files | refused: rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 38
2 files | refused: kv+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 38
2 files | refused: content@open+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 22
2 files | refused: content@write+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 27
2 files | refused: content@sync+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 32
2 files | refused: content@dirsync+rename+rm+mkdir | durable record: yes | old token accepted: no | mint succeeds: no | runs: 37
2 files | refused: kv+content@open+rename+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 24
2 files | refused: kv+content@write+rename+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 29
2 files | refused: kv+content@sync+rename+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 34
2 files | refused: kv+content@dirsync+rename+rm+mkdir | durable record: no | old token accepted: no | mint succeeds: no | runs: 39

runs that accepted an old token or minted (0):

How it is computed:

• Fault classes: kv write (bb.storage.kv.set), content of the three deletion records (the atomic marker with its temporary file, the archive markers, the in-place marker), rename, rm/unlink, and mkdir of the .d directory. Content stays one class, and its refusal is drawn at one point per cell: the open, the write after open, the file sync, or the directory sync. A subset with content is run once per point (content@open, content@write, content@sync, content@dirsync). That gives 16 subsets without content plus 16 × 4 with it, 80 cells per file count, × {1, 2} credential files × both lifetimes. Three separate classes would give 256 subsets per file count, mostly combinations where the earliest refused point already decides every path. Faults apply during the delete and are lifted at the restart, the worst case for minting.
• Runs per cell:
  • one run that completes the delete;
  • one run crashing after every counted mutation of the delete, from before the first one;
  • one run crashing after every counted mutation of the next start, followed by a second start.
  • The write, the file sync and the directory sync are counted mutations, so every crash point between them is a run.
  • Every run first starts a store that has not swept yet (initialize only), mints a thread and a nested credential, then presents both old tokens. That is the window Hermes named in round 12: the first contact after the restart is a mint, with no local record left. The crash restart and the final restart, which sweep, follow it.
• Power loss at every restart. The test tracks durability. A file's content is durable once its file sync succeeds. A created, renamed or mkdir'd entry is durable once a later sync of the credential directory succeeds. Before every restart the test drops every change that is not durable: entries created since the last directory sync are removed, renames since then are reversed, and files with unsynced content are truncated to empty. That is the worst a power loss can leave.
• The lookup stub answers gone after every restart, in every cell and at every crash point, whatever the plugin managed to record. It stands in for bb's thread row: the delete route commits deletedAt before it emits thread.deleted, and nothing clears it. The server deletion matrix below reads that row for real. The kv-refused live branch of rounds 11 and 12 is gone.
• Columns: "old token accepted" and "mint succeeds" are the worst result over every run of the cell. "durable record" is whether the completed delete left a durable record: the kv row, or a filesystem record whose writes and syncs all succeeded.
• Invariant check: before every real rm, unlink or rename in every run, the test checks the disk and kv. A credential file may be removed, or renamed to anything but deleted-pending-, only while a durable record of any kind exists. A deleted-pending- file may be removed only while a durable record of another kind exists. The test asserts zero violations.
• Assertion: the rows are compared with the literal EXPECTED_RESIDUAL, written from the rule rather than from the code's output. The rule: no old token and no mint, in every cell. The "durable record" column still varies, and the invariant still holds with zero violations.
• Test cost: the sync stub is a fault-injecting no-op. It can fail at the file sync and at the directory sync, and it records what became durable instead of calling the real fsync. Product code fsyncs the file and the parent directory, and a record counts only after both succeed. Each run's directory is on tmpfs (/dev/shm) where it exists and is removed when the run ends, and one host-token store is shared by every run. thread-tokens-suspend.test.ts runs in its own vitest project (bb-plugin-account-pool:deletion-residual, sequence.groupOrder: 1), after every other account-pool file, because at 4a175ea the power-set running beside thread-tokens.test.ts pushed one of its tests past 5 s under full load (probe run 38560662490). The power-set tests carry an explicit 600 s timeout, five times the 119.7 s worst case measured then. At this head the power-set takes 36.6 s and 37.7 s in the PR CI (run 38584387035, the fresh run of this head's account-pool input hash), up from 24.7 s at e12a324, because every run now adds an unswept first start. The fresh-neighbour probe at 3e5f38e (run 38570665390: 3e5f38e plus empty files in packages/db and packages/plugin-sdk, so those packages and every dependent run uncached) passed with account-pool 707 of 707. Under that full load the power-set took 100.9 s (durable) and 78.7 s (process), under a sixth of its 600 s timeout. thread-tokens.test.ts took 5.1 s for its 79 tests. The neighbouring @bb/db thread-pruning took 5.8 s and @get-bb/plugin-sdk shared-declaration-emit 10.2 s, against 30 s timeouts. The slowest other account-pool tests are the two upstream-transport connect-timer tests (9.5 s and 7.2 s of 20 s) and "exposes Codex device login over RPC and the two-step CLI" (3.1 s of 5 s, the same in every run, loaded or not).
• Server deletion matrix (apps/server/test/helpers/account-pool-delete-matrix.ts, run by account-pool-delete-matrix-{durable,process}-{1,2}.test.ts in the @bb/server:account-pool-delete-matrix vitest project, sequence.groupOrder: 1, after every other server file). It runs the computed table's cells and crash points through the real stack instead of the store-level stub: every subset of refused plugin kv, content, rename, rm and directory mutations, content refused at its open, write, file sync or directory sync, one and two credential files, both lifetimes, and every crash point of the delete, of the first start after it and of the next start. Each of the four files covers one lifetime and one credential count, 80 cells each, 320 in all, and none is sampled.
  • bb: the server test harness (createTestAppHarness, an in-memory SQLite database from createConnection and migrate; no mocked database). Each run seeds a thread and deletes it through the real DELETE /api/v1/threads/:id route. The route runs markThreadDeleted, then the real thread.deleted payload (toThreadResponseFromThread) goes through setPluginThreadEventEmitter into the real account-pool plugin, loaded in the fake plugin host the harness uses in turn-failed-retry.test.ts. Each run asserts the route answered 200 and exactly one thread.deleted was delivered.
  • Restart: at every start after the delete (the unswept first start, the crash restart and the final start), bb's server is rebuilt from the previous connection's serialize() (createTestAppHarness({ db: createConnection(bytes) })), and the plugin is reloaded (lifecycle.reload, the same kv and data directory) after losing every unsynced change.
  • Lookup: the plugin's bb.sdk.threads.get is the real SDK client (createBbSdk with createHttpTransport), whose fetch is the restarted server's app.fetch. Every lookup after the delete reads the thread row from the restarted database through the real GET /api/v1/threads/:id, which answers 404 thread_not_found. Nothing answers for bb. Each run asserts that the restarted row has deletedAt and that both checked starts asked bb.
  • Faults: the store matrix's injector, scoped to the plugin's data directory. The kv fault refuses the plugin's deleted-thread: writes. Every injected refusal and crash throws the matrix's own InjectedFault class. After a crash point, every write in the directory and every plugin kv write fails, and the crashed injection stays installed until the next start's factory, so nothing the crashed instance does afterwards reaches the disk.
  • Crash points: delete@N, first@N and restart@N are enumerated from the same start of the run without a crash, in the same state. A crash is a process death: once crash N fires, no mutation at index N or later in that injection may succeed, and the crashing operation (the thread.deleted delivery, the first start's reload, or the restart's reload and sweep) must still be running when it fires. Either failure is a crash defect and fails the run; that is what proves the crash ended the world even where the plugin absorbs the EIO. A crash that fires at another mutation or never also fails; a crash point beyond its start's mutations is listed and fails the file (there are none). As a sanity check, the crash signal must carry the one InjectedFault (code EIO, mutation N) that the injector created when crash N fired. Every error a crashed operation raises must be an InjectedFault (or carry it as cause); any other error fails the run. The only other error accepted is the plugin's "could not record the deletion of thread " from thread.deleted, and only when that delete had an injected refusal or crash. The matrix counts and prints those per file.
  • Recovery: after every crash, the next start is what production runs: a fresh plugin instance (lifecycle.reload) over the crashed data directory and the restarted database. The matrix asserts that it is neither the crashed instance nor any replaced one, and that it has not swept. It is probed before its sweep, then the sweep runs, and it is probed again. A replaced instance is never probed. The run without a crash ends the same way: a clean second start that sweeps, then a final start probed before and after its sweep.
  • Checks, at the unswept first start (unless it is the start that crashed) and at the recovery or final start, before and after its sweep: the plugin's own store (the instance the plugin built) refuses forThread and forNested with the plugin's mint refusal (any other rejection is an unexpected error); the env export, with any contribution failure other than that refusal counted as an unexpected error, (resolveProviderEnv("claude-code"), native for the process lifetime) carries no ANTHROPIC_AUTH_TOKEN and no nested token; the old thread token gets 401 on POST /v1/messages and the old nested token 401 on GET /availability; and the invariant check finds zero violations.
  • Tokens before the delete: with two credential files, the env export mints both. With one, the plugin's own store forThread mints the thread token, because the export always mints both. The delete, the lookups and every check after it are the same in both.
  • Result at 173c306 (PR CI run 38625913755, Tests (server), fresh): all 320 cells read "old token accepted: no | mint succeeds: no | export carries a pool credential: no", over 13,652 runs (3,280 in each one-file file and 3,546 in each two-file file). Per one-file file: 1,890 delete, 646 first-start and 664 crash-restart crash points; per two-file file: 2,120, 682 and 664. Every one ended the world at its crash point: the operation was still running when its EIO fired, and no mutation at or after it succeeded. 0 beyond, 0 missed, 0 crash defects, 0 unexpected errors, 0 failing runs and 0 invariant violations in each file. The injected EIO also came out through the plugin's own error path in 0 delete, 480 first-start and 480 restart crash runs per file; the rest the plugin absorbed by design. Each file took 379 s to 554 s, and the server job took 22 minutes of its 30. The verbatim rows are in the green log.
  • Memory: the matrix project runs with --max-old-space-size=1024, and each file asserts that its retained heap after a full GC grows by less than 64 MB from cell 1 to cell 80. At 173c306 (run 38625913755): durable 1 file 142, 161, 179 MB at cells 1, 40 and 80; process 1 file 142, 161, 179 MB; durable 2 files 143, 165, 185 MB; process 2 files 142, 165, 185 MB. The heap limit was 1072 MB and the peak resident size 485 to 539 MB per worker.
• No residual for archived threads. An archived thread's tokens are never accepted, whether the archive event arrived or not, and whether or not bb's lookup answers. The cost is that a live thread gets 401 while bb's in-process thread lookup is failing; its credentials stay stored and the same tokens work again once bb answers.
• ThreadLookup returns live | archived | gone. The sweep restores live, suspends archived and deletes only gone. It covers only threads that hold credentials (the live install has 453 historical archive markers) and looks up archived threads last.

server.ts:

• thread.archived calls suspendThread.
• The environment check and the unarchive handler treat a thread bb reports deleted as gone before looking at its archive time.
• The store receives the bb lookup and the sweep lookup timeout.

Which states lose a credential: only delete (the delete event, or bb reporting the thread gone). Apart from that, a credential ends only through its own expiry (process credentials, 30 days) or a machine-token rotation (after the existing 10-minute grace). No archive, unarchive, crash, marker failure, rename failure or unreachable bb removes one.

Where acceptance waits for bb after a start: every authentication asks bb, so no token is accepted after a start until bb has answered for its thread at least once in that process. When bb hangs after a start, tokens are rejected until it answers, which is a deliberate change from ours, where they were accepted; a thread confirmed earlier in the process keeps working through a bb hiccup.

Unchanged: no wire, protocol or stored-event changes.

Rollback: an older build reads an archived: true marker at the files' epoch and deletes those files, which is the old revoke-on-archive behaviour and fails closed. It ignores suspended-* and deleted-pending-* files and leaves them in place, so it never loads them as credentials.

Docs: accounts-and-routing.md, bb-guide-plugins.md and docs/configuration.md now describe suspend, unarchive and delete. The table realignment in configuration.md comes from oxfmt. The account-pool source and test files were already out of format on ours and are reformatted here.

How you verified

• thread-tokens-suspend.test.ts (run for both durable and process lifetimes):
  • The same tokens are accepted after unarchive, in process and across a restart.
  • A missed unarchive event recovers on the next request.
  • Delete is final even when bb later reports the thread live.
  • A thread archived while the store missed the event is rejected after a restart.
  • A marker that cannot be written renames the files aside with their content intact. An offline restart rejects them; a restart with bb live accepts the same tokens and renames the files back.
  • When the marker writes and every rename fail, the files stay. A restart with bb unreachable rejects the tokens; bb archived still rejects; bb live accepts.
  • When renaming back fails on unarchive, the tokens are accepted in process, rejected after a restart while bb is unreachable, and accepted after a restart with bb live.
  • Crash walks crash after every filesystem mutation of an archive, of an unarchive, and of an archive whose marker write keeps failing. Each then restarts with bb archived or unreachable (nothing accepted), then live (the same tokens accepted). Each walk asserts it reached more than one crash point.
• Hermes round 1 (stale cached confirmation):
  • The new server test confirms a thread live, archives it in bb without delivering the event, and immediately checks both token paths: both return 401 while both credential files remain. Once bb reports the thread live, the same tokens return 200.
  • Store tests check that concurrent authentications for one thread make one lookup and that the next call makes a new one, the same archived-without-event case.
  • Red on 4a68635: 7 failures (log red-prevhead-4a686356e…; the server test with "expected 200 to be 401", plus 6 store tests across both lifetimes).
• Hermes round 2 (fallback accepted an archived thread):
  • Hermes's server test, for both token paths: confirm the thread live, archive it in bb without the event, make every lookup fail. Both paths return 401 and both credential files remain stored. Once bb reports the thread live, the same tokens return 200.
  • The store test (both lifetimes) confirms a thread live, archives it in bb, and makes bb unreachable. Both paths are rejected, the thread's files are unchanged, and exactly one warning is logged, a second only after a minute. Once bb answers live, the same tokens are accepted. A second store test checks that a failed lookup is retried once within the same flight. A third makes the lookup hang past the timeout: the token is rejected, the lookup is not retried, and the warning says "timed out".
  • The periodic-sweep test now expects 401 for both threads while every lookup hangs.
  • Red on ours (2550d6f): 9 failures. Red on 4508e97: 8 failures. Hermes's server test fails on both with "expected 200 to be 401"; the earlier round-1 server test passes on 4508e97.
• Hermes round 3 (restore cleared a suspension that arrived during its awaits):
  • Hermes's case, as a store test (both lifetimes) and a server test: a suspended thread, bb reports it live, both token paths in flight. The restore pauses at its first directory read after the version check. thread.archived is delivered, then the restore resumes. Both paths return 401, the next requests return 401, both credential files remain, and once bb reports live the same tokens return 200.
  • Interleaving walk (both lifetimes) for restore, suspend, remove and admit with a live answer. Each transition is paused at every filesystem call and bb lookup in turn (point 1, 2, … until it completes without reaching the point). At the pause, thread.archived, thread.unarchived or thread.deleted is delivered, then the transition resumes. The walk asserts the final state of the last event:
    • archived: in-flight and next requests 401, the same credential files, then 200 once live;
    • unarchived: 200;
    • deleted: 401 and no credential files.
  • An in-flight request that completed before the event is allowed its earlier answer. bb cannot unarchive a deleted thread, so late archive or unarchive events after a delete must leave it deleted. Each transition asserts it reached more than one pause point.
  • Red on ours (2550d6f): 9 failures. Red on 31f2e85: 5 failures, namely Hermes's server test ("expected [ 200, 200 ] to deeply equal [ 401, 401 ]"), Hermes's store test in both lifetimes, and the admit walk in both lifetimes ("admit paused at point 2, then archived: in flight: expected { token: true, nested: true } …"). On 31f2e85 the restore and suspend walks pass, because the next request asks bb anyway; the stale suspension clear shows through the in-flight admit.
• Hermes round 4 (a stale archived answer re-suspended a thread after unarchive):
  • Hermes's case, as a store test (both lifetimes) and a server test. A lookup starts while the thread is archived and is held. The thread is unarchived, and the restore completes (another request on the nested path returns 200). Then the held archived answer is released. The held request returns 200, the same tokens keep returning 200, the credential files are unchanged, and a store reopened without bb accepts them (on-disk state is live).
  • Stale-answer walk (both lifetimes). For admit's lookup and the sweep's lookup, each holding a live, archived or gone answer, it delivers archived, unarchived or deleted while the answer is held and lets the event complete, then releases the answer. It asserts the state of the last event: in-flight and next requests, credential files, a store reopened without bb, and for archived, the same tokens once live.
  • The round 3 walk now delivers unarchive through unarchiveThread.
  • The existing server test "keeps the same token valid when a stale archived lookup races archive and unarchive events" now counts 4 lookups instead of 3: the stale answer is dropped and bb is asked again.
  • Red on ours (2550d6f): 27 failures. Red on 03ad33e: 10 failures, namely Hermes's server and store tests (both lifetimes), the admit walk holding a stale archived or gone answer ("admit holding a stale archived answer, then unarchived: in flight: expected { token: false, nested: false } to deeply equal { token: true, nested: true }"), and the sweep walk holding a stale gone answer, which deleted the credentials of a thread that had been unarchived ("sweep holding a stale gone answer, then unarchived: next request: expected { token: false, nested: false } …").
• Hermes round 5 (deletion was only a suspension, so a live answer could restore a deleted thread and the env path could mint again):
  • Store test (both lifetimes). A held stale live answer is released after the delete, then bb answers live. The test checks old tokens, settleThread, unarchiveThread, restoreThread, the sweep and both mints, then the same after a restart with bb live. Old tokens are rejected, nothing restores, both mints throw, no credential files remain, and the warning is logged.
  • Server test. A held stale live lookup is on the request path, thread.deleted is delivered, then bb answers live. Both token paths return 401 and env resolution returns no ANTHROPIC_AUTH_TOKEN, including after a plugin reload.
  • Store test (both lifetimes): the deleted-marker write fails, then a restart with bb gone records the marker. The thread stays deleted throughout, including when bb answers live afterwards.
  • Both walks' deleted end state also asserts that settle with bb live is false and that both mints throw.
  • Seven older store tests used removeThread as their archive step, then expected restore and a fresh mint to work. The deletion ones now assert that deletion stays final. The three marker-failure ones (transient read error, corrupt marker, fallback marker) now archive with suspendThread and assert that the same credential works after restore.
  • Red on 7e99612: 25 failures, including the server test ("expected 200 to be 401") and "admit holding a stale live answer, then deleted: settle: expected true to be false". Red on ours (2550d6f): 25 failures.
• Hermes round 6 (a failed deleted-marker write left no record across restarts):
  • Hermes's case as a store test (both lifetimes) and a server test. The deleted-marker write is refused and the thread is deleted. After a restart, the first bb answer is live. Settle refuses, both mints throw, both token paths return 401, env resolution returns no ANTHROPIC_AUTH_TOKEN, and no credential files remain. The archive-marker copy carries the deletion.
  • Store test (both lifetimes): every deletion write is refused. The credential files are all renamed suspended- with the same content set, and in process nothing restores or mints. After a restart with bb reporting gone, the deleted marker is written and the files are removed. Then, with bb live, nothing restores or mints.
  • Red on 29e0337: all 5 new tests fail. Red on ours (2550d6f): 8 failures.
• Hermes round 7 (with every deletion write failing, the suspended- files restored after a restart with bb live):
  • Hermes's case as a store test (both lifetimes): every deletion write fails and renames succeed. After a restart whose first bb answer is live, settle refuses, both mints throw, and both token paths are rejected. The thread's files are all deleted-pending- with the original content set, and no deleted marker exists while writes still fail. Once writes succeed, the next start writes the deleted marker and removes the pending files, and the thread stays deleted then and after another restart.
  • The same case as a server test, with env resolution: both token paths return 401 and env resolution returns no ANTHROPIC_AUTH_TOKEN after the reload. This is the round 6 server test, parametrized to also refuse every deletion write.
  • The round 6 all-writes-fail test (restart with bb gone) stays, now expecting deleted-pending- names.
  • Residual test (both lifetimes): every create, write, rename and delete in the directory is refused. removeThread throws. In process, settle, both mints, both token paths, restore and unarchive all stay refused. The directory's names and contents are byte-identical to before the delete.
  • Red on fe976ae: 5 failures, namely Hermes's store test in both lifetimes ("expected true to be false" on settle), the all-writes-fail test in both lifetimes and the server test's all-writes-fail case ("expected false to be true" on the deleted-pending- names). The residual test passes there, because the behaviour it shows is unchanged. Red on ours (2550d6f): 7 failures.
• Hermes round 8 (a refused rename of the only credential file left no pending file and no record):
  • Hermes's case as a named store test (both lifetimes), "keeps a deleted thread deleted after a restart whose first bb answer is live when every deletion-record write and the rename of its only credential file are refused, because the deletion directory records it". There is one credential file, every deletion-record write and the rename are refused, and rm and mkdir are allowed. After a restart whose first bb answer is live, the old token is rejected, settle refuses, and both mints throw. The .d directory is the only deletion record, and the credential file is removed.
  • The residual power-set test above.
  • The round 7 tests that exercise the deleted-pending- path now also refuse the .d directory, the server test's all-writes-fail case included, since otherwise the directory records the deletion first.
  • Red on edac390 (CI run 38540896167, the new tests overlaid on edac390): exactly the two new tests fail, each in both lifetimes. Hermes's case fails with "expected { hostId: 'host-one', …(3) } to be null" (the old token is accepted). The power-set test fails with a table mismatch.
• Hermes round 9 (with content writes, rename and mkdir refused but rm working, the credential files were removed with no durable record, so after a restart with bb live a mint succeeded):
  • The deletion record now goes first to bb's own database (plugin kv), and nothing is removed before a record is durable (above).
  • Hermes's exact case as a named store test (both lifetimes): "records the deletion in bb's database before removing any file when content writes, renames and the deletion directory are refused but rm works, and accepts no old token and mints nothing after a restart whose first bb answer is live". It asserts that the kv write lands before the first credential removal, that no removal happened without a record, and that after the restart both tokens are rejected, settle is false and both mints throw.
  • "keeps the deleted-pending renames as the only record when bb's database, content writes and the deletion directory are refused, …": the renames are the only record and stay in place through a restart and a sweep with the faults still applied. No old token, no mint.
  • "removes nothing and keeps every file byte-identical when bb's database, content writes, renames and the deletion directory are all refused even though rm works, …": the in-process refusal holds through a sweep, and the directory snapshot is unchanged.
  • "retries an unrecorded deletion on the next sweep, …", "rejects every token and refuses mints while bb's deletion records cannot be loaded at start, accepts a live thread's tokens once a sweep reloads them, and never reads them on authentication", and "removes the credentials of a thread deleted only in bb's database once a sweep reloads the deletion records that failed to load at start".
  • The power-set test above, with five fault classes and crash points from before the first record (in round 10 it ran both bb answers; round 11 derives the answer per run). At every real removal it checks that a record was durable, and asserts zero violations.
  • Server test (server.test.ts), new case "only bb's database could record the deletion and no credential file could be renamed": after the reload both token paths return 401, env resolution returns no ANTHROPIC_AUTH_TOKEN, and the files are gone. The "no deletion record could be written" case now also refuses the kv write. Every case asserts the kv row count.
  • Red on 6cc9862 (CI run 38547457178, the round 10 test files from dfc9e0a overlaid on 6cc9862): 14 failures, 677 passed. They are six of the seven new store tests in both lifetimes, plus the server cases "its deleted marker could not be written" (its new kv-row assertion) and "only bb's database could record the deletion …". Among them, the power-set test reports 84 removals made while no record was durable, and the byte-identical test sees the files removed. The seventh store test ("keeps the deleted-pending renames as the only record …") passes on 6cc9862 too: that head already kept the pending renames as the only record when every filesystem record was refused, and the test pins that behaviour here. Typecheck fails there as expected: the old head has neither DeletionRecords nor the kv parameter.
• Hermes round 10 (the live table allowed an old token and a mint after a crash before the first record, even with nothing refused; that crash comes before the kv write, which was never refused):
  • The test model is corrected. bb's answer after each restart is derived per run: live only if this run's kv write was attempted and refused, gone otherwise. That replaces the two answer tables and the separate pre-record column with one table generated from the rule.
  • The corrected model exposed a window in the code: kv-first left no record when a refused kv write was followed by a crash before the first filesystem record (296 runs per lifetime, run 38551897166). The kv write now comes after every filesystem record and the pending rename ("Record order" above, runs 38550550168, 38551897166 and 38552038836).
  • Hermes's round 9 named test now asserts that kv lands before the first credential removal, rather than that kv is mutation 0.
  • The power-set test shares one host-token store across runs and puts each run's directory on tmpfs. It now takes about 6 s per lifetime, down from about 45 s, so the neighbouring tests are no longer squeezed.
  • Red on dfc9e0a (CI run 38554357367; the round 11 test file overlaid on dfc9e0a): 2 failures, the power-set test in both lifetimes. Its table differs from EXPECTED_RESIDUAL, and it prints 308 runs per lifetime that accepted an old token or minted; 296 of them are in the kv-first window. Every other test passes on dfc9e0a, including Hermes's round 9 case.
• Hermes round 11 (createDeletedRecord swallowed write and sync errors and returned true; the matrix counted a successful open as a record, injected content failures only at open and stubbed sync to succeed; the in-place file and directory fallbacks had no directory sync):
  • Every record path now counts only once durable ("A record counts only once it is durable" above). Load treats partial markers as blocking but unconfirmed ("Partial markers at start").
  • The test model refuses content at the open, the write, the file sync or the directory sync, and loses every unsynced change at each restart ("Computed table").
  • Named tests, both lifetimes, each with bb's database working (bb answers gone after the restart) and refused (bb answers live): "counts no deletion record when the write after open fails on every record path …", "… when the file sync fails …" and "… when the directory sync after the in-place create and after mkdir fails …". Each asserts zero invariant violations and that no old token is accepted and both mints throw after every restart. With kv refused it also asserts: no record is durable and every credential file is still there, as a deleted-pending file. In the write and file-sync cases, a power loss leaves the in-place marker empty and the pending renames in place. A restart that still refuses every record write keeps the empty marker and the pending files and removes nothing. A last restart with no faults records the deletion and removes the files. The directory-sync case with kv refused restarts without a power loss, because with every directory sync and kv refused a power loss is the residual cell of the table.
  • Red on fc84033 (CI run 38559366696; the round 12 test file overlaid on fc84033): 8 failed, 695 passed. The failures are the three kv-refused named cases and the power-set test, in both lifetimes. The named cases see credential files removed while no record is durable. The power-set test fails its first assertion with 338 (durable) and 336 (process) removals made while no record was durable. Its printed table differs from EXPECTED_RESIDUAL in 44 of 160 rows, and it prints 505 (durable) and 504 (process) runs that accepted an old token or minted. The kv-working named cases pass on fc84033 too, because the kv row lands there before any removal.
• Hermes round 12 (the tests derived bb's live answer from the plugin's own kv refusal, although bb commits deletedAt before it emits thread.deleted; forThread checked local state, not bb):
  • Every mint and env export asks bb ("bb is asked before every mint and env export" above), so the residual is gone. The test model answers gone after every restart, and every run mints first on an unswept start ("Computed table").
  • Named store tests, both lifetimes: "refuses a mint after a restart when the lookup stub answers gone, as bb does after a committed delete, and no deletion record survived anywhere" (kv, content, rename and mkdir refused; no durable record and no kv row; a fresh store that has not swept; both mints throw "deleted thread" and both old tokens are rejected; zero violations). And "asks bb before every mint: serves a thread bb reports live as before, refuses and suspends one bb reports archived, and refuses without changing anything while bb cannot answer": a live thread mints with exactly two lookups and returns the same tokens again; unreachable refuses both mints and leaves the directory byte-identical; archived refuses both mints and suspends the tokens; live again returns the same tokens; gone deletes and removes the files.
  • Real-route server test (apps/server/test/threads/account-pool-delete.test.ts): it loads the real account-pool plugin in the fake plugin host, the harness turn-failed-retry.test.ts uses, with bb.sdk.threads.get reading the real thread row from the harness database. The thread is deleted through the real DELETE /api/v1/threads/:id route: markThreadDeleted, then emitPluginThreadDeleted through setPluginThreadEventEmitter, delivered to the plugin as thread.deleted with the real toThreadResponseFromThread payload. The plugin's deleted-thread: kv writes and every filesystem record (marker opens, deleted-pending renames, the .d mkdir) are refused from before the delete through the restart (lifecycle.reload, the same kv and data directory). After the restart the row's deletedAt is set; the env export returns no ANTHROPIC_AUTH_TOKEN and no nested token; the thread token gets 401 on /v1/messages and the nested token 401 on /availability; bb was asked; and there is no deletion record anywhere, in kv or on disk, before or after. This test passes on e12a324 too, because the export path already asked bb there: it is coverage, not the red.
  • Three existing tests change because the mint now asks bb, not because an assertion was weakened:
    • "keeps the same token valid when a stale archived lookup races archive and unarchive events" (server.test.ts) now expects 6 lookups instead of 4: the two new ones are the thread and nested mints asking bb. The same token and the 200 still hold.
    • "suspends what it minted when the thread is archived while its environment resolves" (server.test.ts) is now "mints nothing when the thread is archived while its environment resolves, because the mint asks bb itself". The second lookup is now the mint's own; it sees archived and refuses, so no credential file is written and nothing needs suspending. The test now asserts that no file of that thread exists, which is stronger than asserting a suspension marker.
    • "retries a failed lookup once within the same flight" (thread-tokens-suspend.test.ts) now fails the first lookup after the mint instead of the first lookup overall, which is now the mint's. It still asserts that one authentication makes exactly two lookup calls and succeeds.
  • Red on e12a324 (CI run 38568828320; the round 13 store and server test files overlaid on e12a324): 6 failed, 701 passed. The failures are the two new named store tests and the power-set test, in both lifetimes. "asks bb before every mint …" fails at its first assertion: the two mints made 0 lookups, not 2. "refuses a mint after a restart when the lookup stub answers gone, as bb does after a committed delete, and no deletion record survived anywhere" fails because forThread resolved with a token instead of throwing. The power-set's unswept first start mints in all 160 cells of the table (3420 runs per lifetime mint, and none accepts an old token, because authentication already asked bb there), so every row reads "mint succeeds: yes"; it reports zero invariant violations. The real-route server test passes on e12a324 too (1 test, 204 ms, in run 38566953196 with the same Turbo input hash as this red), because the env export already asked bb there: it is coverage, not the red.
• Hermes round 13 (the power-set's deletionOutcome sets a synthetic bb answer and calls store.removeThread() directly, so the matrix did not answer from bb's real thread row; only the single all-refused case in account-pool-delete.test.ts used the real route):
  • The whole matrix now also runs through the real path ("Server deletion matrix" above): every cell and every crash point, one and two credential files, both lifetimes. The delete goes through DELETE /api/v1/threads/:id and the real thread.deleted payload, and every lookup after the delete is the real SDK reading the thread row of a server rebuilt from the serialized database at each restart. Nothing answers for bb there.
  • The store-level power-set stays. Its title and printed header now say it uses a lookup stub in place of bb's committed thread row. The named store test is now "refuses a mint after a restart when the lookup stub answers gone, as bb does after a committed delete, and no deletion record survived anywhere".
  • Harness traps the server matrix closes:
    • createApp installs its own plugin thread-event emitter, so a restarted harness took over thread.deleted. The matrix reinstalls its emitter before every DELETE, and every run asserts exactly one delivery and every completed run at least one delete mutation, so the matrix cannot pass on bb's lookups alone.
    • vi.spyOn keeps every call's arguments and results until the worker exits. On fs and kv that was about 47 MB per cell (heap after forced GC: 192 MB after cell 1, 1853 MB after cell 36), and the two-file files ran out of heap in CI run 38584387035. The matrix overrides the methods directly, with no call history.
    • On Node 22 (CI runs 22.23.3), AbortSignal keeps composite signals in its internal gcPersistentSignals set. One of them had an Error reason whose captured stack held the fake plugin host's dispose closure, its options and the matrix's lookup closure, and with it every run's restarted server harness. Heap snapshots at cells 2 and 8 showed +774 Hono apps, +8140 drizzle query builders and +3375 SQLite statements, and with a 1 GB heap all four files ran out of memory (run 38595856117). The lookup now goes through a module-level link that is cleared when each run ends, and the matrix stops each plugin service with a string reason. On Node 22.23.3 with the 1 GB cap, the durable two-file file retains 142 MB after a full GC at cell 1, 160 MB at cell 40 and 175 MB at cell 80, with a peak resident size of 436 MB. The packaged bb runs Electron's Node 24.20.0, where this retention does not occur, so the installed app is not affected.
    • The matrix workers run with --max-old-space-size=1024. Each file records its retained heap after a full GC at cells 1, 40 and 80, and fails if it grows by 64 MB or more, so a leak fails the matrix instead of hiding under the default limit.
  • Red on e12a324 (CI run 38604903273; e12a324 plus only the matrix helper, the four matrix files and the server vitest config, at 61f14de): all four matrix files fail at the assertion level, and every other server file passes (391). All 320 cells read "old token accepted: no | mint succeeds: yes | export carries a pool credential: no", with 1726 failing runs in each one-file file and 1806 in each two-file file. In every failing run the restarted database has deletedAt, exactly one thread.deleted was delivered, both starts asked bb, no old token is accepted and the export carries no credential. Only the mint at the unswept first start succeeds, through the plugin's own store forThread and forNested, because the export and authentication already asked bb on e12a324. Zero invariant violations. Its memory stays flat too: 141 MB at cell 1 and 168 to 173 MB at cell 80.
• Hermes round 14 (in the server matrix's restart(crashAt, sweep), the catch swallowed any error once crashAt was set, nothing asserted that the error was the injected fault or that mutations reached crashAt, and after a failed reload the next probe could test the previous instance instead of a start over the crashed state):
  • Injected refusals and crashes now throw the matrix's own InjectedFault class. A crashed start may raise only that (directly, as cause or in an AggregateError); anything else is an unexpected error and fails the run. The injector records the index of the mutation where the crash fired, and each run asserts it is exactly N for delete@N, first@N and restart@N. A crash that fires elsewhere or never fails the run. A crash point beyond its start's mutations is listed per file and fails the file; every file has 0.
  • New crash points: every mutation of the unswept first start after the delete (first@N), so all three starts with crash points are covered: the delete, the first start and the crash restart. The final start never crashes; it is the recovery start under test.
  • After every crash, the next start is a fresh plugin instance over the crashed data directory and the restarted database. The matrix rejects a start that reuses the crashed or any replaced instance or its token stores, and one that swept before its first probe. It probes that instance before its sweep, runs the sweep, and probes again. The previous instance is never probed. The crashed injection stays installed until the next start's factory, so the crashed instance cannot write to the disk after its crash.
  • The probes no longer read any rejection as a refusal. forThread and forNested must reject with the plugin's mint refusal (deleted, archived, or not confirmed live by bb). An env export failure other than that refusal, which the fake host logs and turns into [], is an unexpected error.
  • The plugin's thread.deleted handler reports "Account Pooler could not record the deletion of thread " without the underlying error. The matrix accepts that message only for the deleted thread and only when that delete had an injected refusal or crash. It is counted per file: 2056 in each one-file file and 2142 in each two-file file at 58b44f6. With no injected fault, the message is an unexpected error.
  • Gates kept: 320 cells, 0 failures, 0 violations, 0 unexpected errors, retained heap growth under 64 MB from cell 1 to 80, and the 1 GB cap. The per-file test timeout is now 1,800 s, up from 1,200 s, for the added first-start crash points, the clean second start and the extra probe per run.
  • Red on e12a324 (CI run 38616686352; e12a324 plus only the matrix helper, the four matrix files and the server vitest config from 58b44f6, at 70070bb): all four matrix files fail at the assertion level and every other server file passes (391). All 320 cells read "old token accepted: no | mint succeeds: yes | export carries a pool credential: no", with 1,810 failing runs in each one-file file and 1,890 in each two-file file (7,400 in all: 56 runs without a crash, 6,336 delete@N, 336 first@N and 672 restart@N). In every failing run the crash point was reached at exactly its mutation, there are 0 unexpected errors, the restarted database has deletedAt, exactly one thread.deleted was delivered, every probe asked bb, no old token is accepted and the export carries no credential. Only the mint through the plugin's own store forThread or forNested succeeds, because the export and authentication already asked bb on e12a324. 0 crash points missed or beyond, 0 invariant violations, and retained heap 141 to 142 MB at cell 1 and 175 to 180 MB at cell 80, with no OOM.
• Hermes round 15 (expectInjected failed on unexpected errors but accepted an empty error list, so a crash run counted as "reached" even if its injected EIO was swallowed and never surfaced):

  • Design decision. The plugin absorbs write failures on purpose:

    • recordDeletion in thread-tokens.ts tries the deleted file, both archive markers, a second deleted file, createDeletedRecord, createDeletedDirectory, the held-aside files and the kv row in turn, and maps each attempt's error to false or an empty catch;
    • createDeletedRecord and createDeletedDirectory return false on any error;
    • once the deletion is filed, removeProviderFiles(...).catch(() => undefined) absorbs cleanup failures.

    So for many crash points no error leaves the plugin at all, and making it surface every absorbed EIO would be a resilience regression: a delete or a start would fail on a write it is built to survive. The property under test is therefore "a crash is a process death, and refusal holds after it", not "the EIO propagates out of the plugin".

  • The proof that the crash ended the world. Once crash N fires, no mutation at index N or later in that injection may succeed, and the crashing operation must still be running when the crash fires. The operation is the thread.deleted delivery, the first start's reload, or the restart's reload and sweep. Either failure is a crash defect and fails the run. A swallowed crash that lets the write at N or a later one proceed fails here.

  • Sanity check, not proof. The injector makes one InjectedFault (code EIO, mutation N) when crash N fires and rejects the injection's crash signal with it. Each run asserts that the signal carried exactly that instance. Later writes after the crash throw separate InjectedFault objects. The dead operation is awaited to completion with the crashed injection still installed before the next start, and anything it raises must be an InjectedFault.

  • Printed per file: crash runs that ended the world at their crash point, per start, against the number exercised; crash runs whose EIO also came out through the plugin's own error path, as evidence of how often the plugin absorbs it; and the crash defects, all 0.

  • Mutants, each run as a full process, one-file matrix with only the helper changed, logs kept:

    • swallowed crash (the injector marks crash N reached and signals it, but swallows its own throw, so mutation N and later ones proceed): fails at the assertion level through the disk-frozen check, with 3,026 crash defects such as "mutation 0 (open deleted-thread-….json….tmp) succeeded after the crash".
    • no signal (the injector throws the crash without rejecting the crash signal): fails at the assertion level, with 6,400 crash defects ("did not end with the injected EIO crash" and "settled before its crash fired").
    • detached delivery (the harness stops waiting for the crashing thread.deleted handler and records an empty delivery): fails at the assertion level on expect(first.deleteMutations.length).toBeGreaterThan(0), because the delete never reaches the plugin inside its injection, so no crash point is enumerated.
    • detached sweep (the harness stops awaiting the restart's sweep): survives, and is equivalent. The sweep after a delete makes no mutations, so the restart has the same 664 crash points with and without it; every restart crash point is in the reload.
    • A first "no wait" mutant (settle returned without waiting for any operation) broke the harness flow with an Error before any assertion, so it is discarded as ill-formed.
      The log is /tmp/p7-196/mutant-red-r16.log.

  • Red on e12a324 (CI run 38625914382; e12a324 plus only the matrix helper, the four matrix files and the server vitest config from 173c306, at b4f6990): all four matrix files fail at the assertion level and every other server file passes (391). All 320 cells read "old token accepted: no | mint succeeds: yes | export carries a pool credential: no", with 1,810 failing runs in each one-file file and 1,890 in each two-file file (7,400 in all: 56 runs without a crash, 6,336 delete@N, 336 first@N and 672 restart@N). Every failing run has 0 crash defects (its crash ended the world at its mutation) and 0 unexpected errors; the restarted database has deletedAt, exactly one thread.deleted was delivered, every probe asked bb, no old token is accepted and the export carries no credential. Only the mint through the plugin's own store forThread or forNested succeeds. 0 violations, no OOM, retained heap 142 MB at cell 1 and 175 to 181 MB at cell 80.

• Earlier red proofs (logs start with the commit sha):
  • The first 14 tests fail 12 times on ours, with archive mapped to the old removeThread.
  • The marker-failure test fails on d25e8f1.
  • The rename and unconfirmed-thread tests fail 8 times on d741f90.
• Server and store tests that asserted "an archived token never comes back" now assert suspension, the same token after unarchive, and revocation on delete. The fake bb in server.test.ts now reports archived and deleted state consistently with the events it emits. Server tests now expect rejection while bb hangs or fails, and acceptance once it answers.
• An independent adversarial review of d25e8f1 found the marker-failure revocation gap, the archived-before-deleted ordering and the sweep ordering, all fixed here. The coordinator's pre-review of d741f90 found the credential loss when files were deleted on marker failure, replaced by the rename-aside path.
• Green at 173c306 (this head; ours 394a006 is an ancestor): the PR's CI run 38625913755 passed (10 jobs, 2 skipped), including typecheck, lint and oxfmt --check. @bb/server ran fresh (cache miss, input hash d1b22fad94276f70): 396 files passed and 5 skipped (401), 3954 tests passed and 3 skipped (3957), including account-pool-delete.test.ts (1 test, 280 ms) and the four server deletion matrix files (durable 1 file 498 s, process 1 file 499 s, durable 2 files 554 s, process 2 files 379 s), every one under the 1 GB heap cap and with retained heap flat as listed above. account-pool: 16 files, 707 tests passed (a cache hit of input hash 8f68d6b4fd7e2162, run fresh in the PR's CI run 38584387035 at 5efd26e, whose plugins/account-pool tree is identical); there the power-set test took 36.6 s (durable) and 37.7 s (process), and the whole suspend file 78.7 s.

Supersedes #193.

AGENT GENERATED