Skip to content

feat: share a lease among the background jobs, start them over after a restore, and report their status - #260

Merged
benatespina merged 19 commits into
mainfrom
feat/personal-data-operations
Nov 9, 2026
Merged

benatespina merged 19 commits into
mainfrom
feat/personal-data-operations

Conversation

@benatespina

@benatespina benatespina commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

What

Operating the runtime's background work, for personal data and dead letters, as one:

  • Shared lease. The backfill, the key rotation, the retention and the purge of resolved dead letters each run in one instance at a time, under a lease kept in bounda_checkpoints (kernel/shared/lease.ts, extracted from the backfill's). The lease is taken again before each item, so an instance that stalled past it stops at the next one and the instance that took it over goes on. Instances that do not hold it read the lease row, not the work. worker.stop() gives every lease back.
  • Restores while running. Every instance moves runtime:generation forward at most every five seconds and remembers what it read. The first instance to find it went back counts the restore in runtime:restores, which every other instance then sees change (kernel/shared/restores.ts). On a restore the forget ledger is reconciled in the same pass, every job forgets what it knew (backfill caught up, rotation complete, pauses, the dispatcher's backoff), and the first instance clears the holds the copy brought back (PersonStore.releaseHolds, PersonalKeyStore.releaseHolds, the backfill's and the ledger's rows). Handler attempts stay: those runs happened.
  • Restores while the app was stopped, the web role, and no writes at rest. ForgetLedger.readMark/raiseMark are new; marks only go up. They live in memory, in SQL (forget-ledger migration 2, <prefix>forget_ledger_marks) and in KV. They keep the generation outside the database: the mark is raised with every move of the generation (at most every five seconds, after activity), and on a stop after any write. A forgetting the ledger holds that the database does not also counts as a restore, whatever the generation says. A start that finds the database's generation below the mark has found a copy restored while the app was stopped, and starts the jobs over before serving. Without a ledger, bounda restored (app.restored()) declares the restore. An app in the web role runs only a restore watch, which reconciles the ledger at once on a find. The generation moves only after activity, a write to the stream or a worker pass that did work, so an idle instance writes nothing. The guide's "Restores" table says what each path guarantees.
  • Person index. Already backfilled at start by migration 1 of feat: version the runtime's tables and migrate them forward on start #254 and pinned by storageSchemaContract; nothing to add.
  • ProcessAbandoned. Discarding a process dead letter ends its instance with ProcessAbandoned { letterId } in the same transaction, acted by whoever discards. ProcessStatus gains "abandoned". Parking no longer reads the letter, so the purge of resolved letters never decides whether an instance was given up. An instance discarded before this release is abandoned on the next event that reaches it.
  • Each subscriber's last failure. The new table bounda_subscriber_failures is storage migration 2 on SQLite, libSQL, the Durable Object and PostgreSQL. It keeps one row per subscriber, with the text sealed under the person the event names whatever runtime.telemetry.errorText says. client.subscriberFailures.list()/get() open it and are recorded in the access log (subscriberFailures.list, subscriberFailures.get). connect and React Router's Cloudflare clients expose it too, and the export includes it (art. 15).
  • bounda status and bounda migrate. client.status() reports, by database, each table scope's version and the latest this release migrates to, the restores found, every subscriber's lag and last failure without its text, and the backfill, ledger, rotation, retention and purge. Adapter.schemaVersions(), Adapter.migrate(scopes) and migrateTables({ config }) are new. bounda status (--json, --error-text through the recorded call) prints it. bounda migrate migrates every database the configuration names before a deploy, reading only the configuration.

subscriber_failures is an inspected store (InspectedStore), so expectForgotten (#253) searches it, with its own advice.

Breaking for custom adapters and ledgers (minor): Storage.subscriberFailures, Adapter.schemaVersions/migrate, ForgetLedger.readMark/raiseMark, and the new person and key store methods.

Tests

  • Lease and restore watch run through the checkpoint store contract on memory, node:sqlite, libSQL, PostgreSQL and workerd: two instances taking the lease at once, a lease that lapses mid-way, a lease given back, a restore found by every instance with exactly one first. The subscriber failure store contract and the new person and key store cases run on all five.
  • Rotation, retention and purge: two instances, a lease that lapses mid-way with no key rewrapped and no person forgotten twice, and a restore taken mid-rotation that brings back a held key.
  • A hot restore on node:sqlite while the app runs: every bounda_* table is copied, then deleted and inserted back. It covers the rotation starting over without the copy's holds, the ledger forgetting again at once, and two instances, one of them first.
  • ProcessAbandoned: fold, discard with actor, a discard a retry settled, a letter gone without a discard, and instances discarded before the release.
  • storageSchemaContract covers schemaVersions/migrate on every adapter. The app tests cover failures sealed, opened, forgotten and exported on memory and SQLite. The CLI tests cover status and migrate, and test-types covers the new types.

Checks

  • pnpm check with Docker: lint, build, generate, package and example typecheck, and tests are green after rebasing on feat(core): expectForgotten checks nothing of a forgotten person is left #253 (3518 passed; 4 timeouts under load in the generator and the Vite plugin, untouched here, pass alone). astro check of docs fails locally with this.bridge.deleteEnv is not a function, the same with main's docs (follow-up W1).
  • Stryker on changed code: core 91.08 % (the restore additions 91.76 %), cli 96.83 %, postgresql 96.67 %, cloudflare 92.55 %, react-router 100 %. The surviving mutants are pass-throughs no path calls, the shortcut for instances that ended, and span names.
  • /code-review high, twice: the second, on the restore additions, found a missing JSDoc (fixed) and a race on KV's mark that the next raise corrects (kept, documented). The first: 10 found. 8 fixed, 1 not a defect: sealing under the store's person would lose the text of a person without a key. 1 kept: the restore watch's write every five seconds, which detection needs, listed as a follow-up.

Left out

@benatespina
benatespina force-pushed the feat/personal-data-operations branch from cda6f4c to a575eb1 Compare November 9, 2026 20:31
… the web role, and move the generation only after activity
…writes, and take a forgetting the database lost for a restore
@benatespina
benatespina force-pushed the feat/personal-data-operations branch from e9a4632 to 567fddc Compare November 9, 2026 21:49
@benatespina
benatespina merged commit e5a837c into main Nov 9, 2026
10 checks passed
@benatespina
benatespina deleted the feat/personal-data-operations branch November 9, 2026 22:56
benatespina added a commit that referenced this pull request Nov 10, 2026
Integrates B4 (#260), A2b (#259) and B2 (#261): the forget ledger's intention
carries the categories a forgetting asks for and the reconciliation commits
again only those; the categories migrations follow B4's and B2's in each scope;
bounda status reports each retention.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant