Skip to content

feat: find a restore by the newest event seen, before anything is delivered, and test restores end to end on every engine - #258

Merged
benatespina merged 6 commits into
mainfrom
feat/restore-harness
Nov 10, 2026
Merged

benatespina merged 6 commits into
mainfrom
feat/restore-harness

Conversation

@benatespina

@benatespina benatespina commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

What

Built on #260 (B4's restore watch), with #261 (B2) and #263 (B1) merged in, as agreed between the two pieces: one detection module, B4's.

  • Newest-event witness in createRestoreWatch. Every instance also watches the newest event it read, or committed as a forgetting (saw(), fed by withCommittedEvents with every event the instance commits), at its position. Events are only ever appended, so a copy that lost it is found at the next check, even one taken in the five seconds before the generation moved, or right after a forgetting; without it, the web and "right after a forgetting" cases of the harness fail. Positions are reused after a restore, hence the id. Concurrent checks share one reading; an active caller that joins one counts at the next.
  • Nothing delivered or claimed before. The dispatcher checks at the start of every pass (beforePass, inside its lock) and, on a find, starts the jobs over (worker.restored) and reconciles the ledger before it delivers; the worker's pass that finds one reconciles before it claims anything, so a scheduled command of someone the copy brought back never runs. worker.restored takes the finding and skips one already handled, so a check shared by the dispatcher and the worker restarts once. The web timer catches the read models up after it forgets someone. Without a ledger, a find logs an error that the persons forgotten since are back.
  • restoreContract in @bounda-dev/core/adapter/testing: persons write events, a read model row, a snapshot, an idempotency record of a rejection that quotes them, a scheduled command, a dead letter and a process; a backup is taken; some are forgotten, the master key rotates, more is written; the copy is restored with the app stopped (found by the ledger's mark) and, where the engine can, running. Every person forgotten after the copy is checked with expectForgotten over the adapter's inspect and over what the app reads back (query, export, dead letters); their scheduled commands never run; everyone else reads under the previous master key and is rewrapped; the app goes on; restoresFound counts one; a failed reconciliation in the copy does not hold the restored start. Also: an instance in the web role alone; a database without a ledger through app.restored() and forgetAgain; an intention the ledger recorded whose commit never happened (feat: record every forgetting in the forget ledger before it commits #261), committed at the restored start under its own id; a strict ledger that cannot be read, which fails the restored start, after which the next start applies the whole restore; and a forgetting of some categories alone beside one of a person as a whole (feat: categories of personal data, each with its own key, basis and retention #263), where only those categories go again and what a legal obligation keeps stays readable. A forgetting the copy lost comes back under its own id, time and reason, as feat: record every forgetting in the forget ledger before it commits #261 records it.
  • Runs on node:sqlite (backup API in place, hot), libSQL (hot, in one write batch; cold, beside), PostgreSQL (pg_dump --format custom, pg_restore --clean --if-exists --single-transaction, hot, testcontainers) and the Durable Object (workerd has no point-in-time recovery, so a copy of every table, index and trigger restored in one transactionSync, then a new app).
  • Docs: new guide "Backups and restores" (steps per engine, what is guaranteed, the contract for adapter authors); the restores table and paragraph in the personal-data guide and the deployment page updated.

Checks

  • pnpm check with Docker: lint, build, generate, typecheck of packages and examples, 3593 tests passed, 11 skipped. astro check of docs crashes locally in @emnapi/runtime before checking any file (Node 24 and 25 alike); CI runs it.
  • Stryker on the changed code of core: own lines of restores.ts and worker.ts 100 %; ledger-append.ts 100 %; app.ts left with equivalent mutants only.
  • /code-review high, twice: first round 6 found, 3 fixed; after the rebase onto feat: share a lease among the background jobs, start them over after a restore, and report their status #260, 4 found, 3 fixed (activity of a joined check, a misplaced comment, a closure used before its declaration), 1 kept (the check on every dispatcher pass is the cost of delivering nothing before a restore is handled).

Left out

  • Read models in another database (readModels: { name: adapter }) restored from another moment: their checkpoint may stand ahead of the stream, so forgettings applied again at reused positions do not reach them. Rebuild before serving: next piece.
  • strict under a running app: a request between the restore and the next pass reads the copy. Check per request: next piece.
  • A bounda restore that restores libSQL in place through the app's own engine (SQLite's backup API from another library corrupted a file libSQL had open): next piece.

…ivered, and test restores end to end on every engine

The restore watch also watches the newest event each instance read, or
committed as a forgetting, at its position: a copy that lost it is found
at the next check, even one taken in the five seconds before the
generation moved, and in the web role. The dispatcher checks at the start
of every pass, and a pass that finds a restore reconciles the forget
ledger before it delivers or claims anything.

restoreContract in @bounda-dev/core/adapter/testing restores a backup
with the app stopped and, where the engine can, running, after
forgetting, key rotation and more writes, also under a web instance and
without a ledger, and checks every person forgotten after the copy with
expectForgotten. It runs on node:sqlite, libSQL, PostgreSQL and the
Durable Object. The new "Backups and restores" guide has the steps.
@benatespina benatespina changed the title feat: notice a restore under a running app and test restores end to end on every engine feat: find a restore by the newest event seen, before anything is delivered, and test restores end to end on every engine Nov 9, 2026
@benatespina
benatespina merged commit 0f02f97 into main Nov 10, 2026
8 checks passed
@benatespina
benatespina deleted the feat/restore-harness branch November 10, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant