Repository navigation
feat: find a restore by the newest event seen, before anything is delivered, and test restores end to end on every engine - #258
Merged
Conversation
benatespina
force-pushed
the
feat/restore-harness
branch
from
November 9, 2026 20:15
84501ec to
1a1becf
Compare
…ivered, and test restores end to end on every engine The restore watch also watches the newest event each instance read, or committed as a forgetting, at its position: a copy that lost it is found at the next check, even one taken in the five seconds before the generation moved, and in the web role. The dispatcher checks at the start of every pass, and a pass that finds a restore reconciles the forget ledger before it delivers or claims anything. restoreContract in @bounda-dev/core/adapter/testing restores a backup with the app stopped and, where the engine can, running, after forgetting, key rotation and more writes, also under a web instance and without a ledger, and checks every person forgotten after the copy with expectForgotten. It runs on node:sqlite, libSQL, PostgreSQL and the Durable Object. The new "Backups and restores" guide has the steps.
benatespina
force-pushed
the
feat/restore-harness
branch
from
November 9, 2026 23:15
1a1becf to
cd6bfd7
Compare
…fore removing their files
…g and legal retention across a restore
… file databases on Windows
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Built on #260 (B4's restore watch), with #261 (B2) and #263 (B1) merged in, as agreed between the two pieces: one detection module, B4's.
createRestoreWatch. Every instance also watches the newest event it read, or committed as a forgetting (saw(), fed bywithCommittedEventswith every event the instance commits), at its position. Events are only ever appended, so a copy that lost it is found at the next check, even one taken in the five seconds before the generation moved, or right after a forgetting; without it, theweband "right after a forgetting" cases of the harness fail. Positions are reused after a restore, hence the id. Concurrent checks share one reading; an active caller that joins one counts at the next.beforePass, inside its lock) and, on a find, starts the jobs over (worker.restored) and reconciles the ledger before it delivers; the worker's pass that finds one reconciles before it claims anything, so a scheduled command of someone the copy brought back never runs.worker.restoredtakes the finding and skips one already handled, so a check shared by the dispatcher and the worker restarts once. Thewebtimer catches the read models up after it forgets someone. Without a ledger, a find logs an error that the persons forgotten since are back.restoreContractin@bounda-dev/core/adapter/testing: persons write events, a read model row, a snapshot, an idempotency record of a rejection that quotes them, a scheduled command, a dead letter and a process; a backup is taken; some are forgotten, the master key rotates, more is written; the copy is restored with the app stopped (found by the ledger's mark) and, where the engine can, running. Every person forgotten after the copy is checked withexpectForgottenover the adapter'sinspectand over what the app reads back (query, export, dead letters); their scheduled commands never run; everyone else reads under the previous master key and is rewrapped; the app goes on;restoresFoundcounts one; a failed reconciliation in the copy does not hold the restored start. Also: an instance in thewebrole alone; a database without a ledger throughapp.restored()andforgetAgain; an intention the ledger recorded whose commit never happened (feat: record every forgetting in the forget ledger before it commits #261), committed at the restored start under its own id; a strict ledger that cannot be read, which fails the restored start, after which the next start applies the whole restore; and a forgetting of some categories alone beside one of a person as a whole (feat: categories of personal data, each with its own key, basis and retention #263), where only those categories go again and what a legal obligation keeps stays readable. A forgetting the copy lost comes back under its own id, time and reason, as feat: record every forgetting in the forget ledger before it commits #261 records it.pg_dump --format custom,pg_restore --clean --if-exists --single-transaction, hot, testcontainers) and the Durable Object (workerd has no point-in-time recovery, so a copy of every table, index and trigger restored in onetransactionSync, then a new app).Checks
pnpm checkwith Docker: lint, build, generate, typecheck of packages and examples, 3593 tests passed, 11 skipped.astro checkofdocscrashes locally in@emnapi/runtimebefore checking any file (Node 24 and 25 alike); CI runs it.core: own lines ofrestores.tsandworker.ts100 %;ledger-append.ts100 %;app.tsleft with equivalent mutants only./code-reviewhigh, twice: first round 6 found, 3 fixed; after the rebase onto feat: share a lease among the background jobs, start them over after a restore, and report their status #260, 4 found, 3 fixed (activity of a joined check, a misplaced comment, a closure used before its declaration), 1 kept (the check on every dispatcher pass is the cost of delivering nothing before a restore is handled).Left out
readModels: { name: adapter }) restored from another moment: their checkpoint may stand ahead of the stream, so forgettings applied again at reused positions do not reach them. Rebuild before serving: next piece.strictunder a running app: a request between the restore and the next pass reads the copy. Check per request: next piece.bounda restorethat restores libSQL in place through the app's own engine (SQLite's backup API from another library corrupted a file libSQL had open): next piece.