feat: find a restore by the newest event seen, before anything is delivered, and test restores end to end on every engine

.changeset/restore-harness.md

@@ -0,0 +1,5 @@
+---
+"@bounda-dev/core": minor
+---
+
+The restore watch also watches the newest event each instance read, or committed as a forgetting, at its position: a copy that lost it is found at the next pass, even one taken in the five seconds before the generation moved, and an instance in the `web` role finds it as well. The dispatcher checks at the start of every pass, and a pass that finds a restore reconciles the forget ledger before anything is delivered or claimed, so a scheduled command of someone the copy brought back never runs. An app without a forget ledger logs an error when it finds a restore. `restoreContract` in `@bounda-dev/core/adapter/testing` is the end-to-end restore an adapter has to pass: persons write, a backup is taken, some are forgotten, the master key rotates, and the copy is restored with the app stopped and, when the engine can, running, also under an instance in the `web` role, without a forget ledger through `app.restored()`, with a forgetting the ledger recorded whose commit never happened, with a strict ledger that cannot be read at the restored start, and with a forgetting of some categories alone beside one of a person as a whole, where only those categories go again and what a legal obligation keeps stays readable; `expectForgotten` checks every person forgotten after the copy, and the app goes on. It runs on node:sqlite, libSQL, PostgreSQL and the Durable Object. The new guide "Backups and restores" has the steps for each engine and what is guaranteed.

docs/src/content/docs/guides/backups.md

@@ -0,0 +1,135 @@
+---
+title: Backups and restores
+description: Taking a backup, restoring it with the app stopped or running, and what Bounda guarantees about the persons forgotten in between.
+sidebar:
+  order: 12
+---
+
+A backup is a copy of the database at one moment. Restoring it takes every table back to that
+moment, the keys of the persons forgotten since included, so their data would open again. With a
+[forget ledger](/guides/personal-data/#the-forget-ledger), a record of every forgetting kept
+outside the database, Bounda forgets them again. This page is what an operator does, and what the
+runtime then guarantees.
+
+## Before you need one
+
+- Configure a [forget ledger](/guides/personal-data/#the-forget-ledger) in a store the restore
+  does not touch, and set its `keepFor` longer than your oldest backup.
+- Keep every [master key](/guides/personal-data/#rotating-the-master-key) a backup was taken under
+  as long as the backup: a backup restored after a rotation opens only with the old key in
+  `previous`, and the worker rewraps its keys under the current one again.
+- Keep the read models that live in [another database](/reference/configuration/) on the same
+  schedule as the storage, and restore both to the same moment.
+
+## Taking a backup
+
+Take it with the app running, with the engine's own tools: they copy a consistent snapshot while
+the app writes.
+
+- **SQLite.** `VACUUM INTO '/backups/app.db'`, or the `.backup` command of the `sqlite3` shell.
+- **PostgreSQL.** `pg_dump --format custom`, of the schema the adapter uses, or the provider's
+  point-in-time recovery.
+- **Durable Object.** Nothing to do: Cloudflare keeps 30 days of point-in-time history of every
+  object's storage.
+
+## Restoring
+
+At start, stop the app, put the copy back and start it:
+
+- **SQLite.** Copy the backup to the database's path, or next to it and point `path` at it.
+- **PostgreSQL.** `pg_restore`, or a point-in-time restore to another server and `url` pointed at
+  it.
+- **Durable Object.** `ctx.storage.onNextSessionRestoreBookmark(bookmark)` with a bookmark from
+  `getBookmarkForTime`, then `ctx.abort()`: the object restarts on the restored storage.
+
+The app finds the restore before it serves anything, by the forget ledger's mark, reads the
+ledger, forgets again whoever the copy brought back, and the read models hear of it before the
+start resolves. A ledger that does not answer within ten seconds is left to finish in the
+background, unless the ledger is [strict](/guides/personal-data/#what-the-runtime-does-with-it).
+Without a forget ledger nothing outside the database says it went back: run `bounda restored`,
+then [`bounda forget --from`](/reference/cli/#bounda-forgotten-and-bounda-forget) with the list of
+whom you forgot since the copy, saved before the restore.
+
+Under a running app, restore in place, in one transaction, so the app sees the database as it was
+or as the copy has it, never half restored:
+
+- **SQLite.** In one write transaction on the app's own engine. Stop the app first to restore a
+  libSQL database with the `sqlite3` shell's `.restore`: SQLite's backup API, run by another
+  library on a file libSQL has open, left it corrupted in our tests.
+- **PostgreSQL.** `pg_restore --clean --if-exists --single-transaction --schema <schema>`.
+
+Every instance finds it on its next pass, before the pass delivers any event or claims any
+scheduled command, and an instance in the `web` role within five seconds;
+[restores](/guides/personal-data/#restores) says how. It logs
+
+```
+the database went back to an earlier copy; background jobs start over
+```
+
+and reads the ledger at once. A request served between the restore and that pass reads the copy as
+it is. A Durable Object has no running app to restore under: restoring its storage restarts it.
+An app without a forget ledger logs an error as well, since it has no record of whom to forget.
+
+## What is guaranteed
+
+Bounda's restore tests run this on SQLite (node:sqlite and libSQL), PostgreSQL and the Durable
+Object, with the app stopped and, where the engine allows it, running: persons write their data,
+a backup is taken, some are forgotten, the master key rotates, more is written, and the copy is
+restored.
+
+- **Whoever was forgotten after the copy is forgotten again,** their `PersonalDataForgotten`
+  committed once more under the id, time and reason the ledger recorded before it first committed:
+  no key,
+  snapshot, idempotency record of a rejection or scheduled command of theirs, no row a projection
+  of `PersonalDataForgotten` removes, their fields `null` in the export, in every event and dead
+  letter and in the state of their processes, and none of their values in any row of the
+  database.
+- **Their scheduled commands never run**, even when they fall due as the copy comes back.
+- **A forgetting of some categories alone** goes again for those categories alone, and what a
+  [legal obligation](/guides/personal-data/#categories) keeps after a person is forgotten stays readable.
+- **A forgetting whose commit never happened**, recorded in the ledger by a process that stopped
+  before committing it, is committed at the restored start too.
+- **A strict ledger that cannot be read** fails the restored start before it serves anything; the
+  next start, once it can, applies the whole restore.
+- **Whoever was forgotten before the copy stays forgotten,** once: the copy holds their forgetting.
+  Someone who registered after the copy is gone with it, and their entry in the ledger finds no
+  key to remove.
+- **Everyone else reads as before,** their keys wrapped under the previous master key open and are
+  rewrapped under the current one.
+- **The app goes on:** projections stay current, new persons register, `forget` and the rotation
+  work, and the next normal start forgets nobody again. A failed reconciliation the copy
+  remembers does not hold the restored start back, and `bounda status` counts the restore.
+- **Without a forget ledger,** `bounda restored` counts the restore and starts the jobs over, and
+  `bounda forget --from` forgets again whom you list.
+
+What ran after the copy was taken runs again: scheduled commands of persons nobody forgot, and
+reactions to events the copy holds but its subscribers had not reached yet. Effects outside the
+database, such as an email, happen twice unless their receiver deduplicates them.
+
+## For adapter authors
+
+`restoreContract` from `@bounda-dev/core/adapter/testing` is the scenario above. Give it a fresh
+database, a ledger store apart from it, and the engine's backup and restore:
+
+```ts
+import { restoreContract } from "@bounda-dev/core/adapter/testing";
+
+describe("my adapter", () => {
+  restoreContract({
+    hot: true,
+    create: async () => ({
+      storage: myAdapter({ url }),
+      forgetLedger: myAdapter({ url: ledgerUrl }),
+      backUp: () => takeBackup(),
+      restore: async () => {
+        await putTheBackupBack();
+        return myAdapter({ url });
+      },
+    }),
+  });
+});
+```
+
+`hot` says whether the engine restores in place under a running app. Each forgotten person is
+checked with [`expectForgotten`](/guides/personal-data/#in-tests) over the adapter's `inspect`, every table
+under its prefix, and over what the app reads back of them.

docs/src/content/docs/guides/deployment.mdx

@@ -363,7 +363,8 @@ them again when the app starts on the restored copy, or at once when the copy is
 running app: every running instance finds that the database went back, and its background jobs
 start over (see [restores](/guides/personal-data/#restores)). Restored while the app was stopped,
 the ledger tells it at the next start, and without a ledger run `bounda restored`. Set the
-ledger's `keepFor` longer than the oldest backup you keep.
+ledger's `keepFor` longer than the oldest backup you keep. [Backups and restores](/guides/backups/)
+has the steps for each engine.
 
 ## Observability
 

docs/src/content/docs/guides/personal-data.md

@@ -1230,14 +1230,19 @@ and `bounda status` counts the restores found. How it is found depends on when t
 
 | The copy was restored | Found by | What is guaranteed |
 | --- | --- | --- |
-| under the running app | every instance, on its next pass, or within five seconds in the `web` role | Any copy taken before the last write the instances saw. A copy taken after it differs from the database in nothing but waits, which the jobs read again anyway. |
+| under the running app | every instance, on its next pass, before it delivers an event or claims a scheduled command, or within five seconds in the `web` role | Any copy taken before the last write the instances saw, or before the newest event an instance read or committed. A copy taken after both differs from the database in nothing but waits, which the jobs read again anyway. |
 | while the app was stopped, with a forget ledger | the first instance to start, before it serves anything | Any copy taken before the last mark the ledger holds, which follows every move of the counter and a stop after any write, or before a forgetting the ledger holds and the copy does not. Only after a crash can a copy of the last five seconds of writes, with no forgetting among them, go unseen: it differs from the database in waits alone. |
 | while the app was stopped, without a ledger | `bounda restored`, which you run after restoring | What you declare: the database alone cannot tell a copy from itself. |
 
 Under the running app, every instance moves a counter in `bounda_checkpoints` one forward after
 it wrote, at most every five seconds, and remembers what it read: a copy holds less, and the
 first instance to see it counts the restore in another row, which every other instance then sees
-changed. An idle instance writes nothing, so neither does its watch. With a forget ledger the
+changed. An idle instance writes nothing, so neither does its watch. Every instance also watches
+the newest event it read, or committed as a forgetting, at its position: events are only ever
+appended, so a copy that lost it is found at once, even one taken in the five seconds before the
+counter moved. A pass that finds a restore reconciles the ledger before the dispatcher delivers
+anything or the worker claims anything, so a scheduled command of someone the copy brought back
+never runs. [Backups and restores](/guides/backups/) has the steps for each engine. With a forget ledger the
 counter is also raised in the ledger, as a mark outside the database, which is how a start tells
 a copy restored while the app was stopped; and a forgetting the ledger holds that the database
 does not is a restore too, whatever the counter says, so the forgettings a copy undid are

packages/cloudflare/test/restore.test.ts

@@ -0,0 +1,101 @@
+import { runInDurableObject } from "cloudflare:test";
+import { env } from "cloudflare:workers";
+import { restoreContract } from "@bounda-dev/core/adapter/testing";
+import { defineForgetLedger } from "@bounda-dev/core/config";
+import { describe } from "vitest";
+import { durableObjectAdapter } from "../src/adapter.ts";
+import { createKvForgetLedger } from "../src/forget-ledger.ts";
+import { insideObject } from "./inside-object.ts";
+
+interface SchemaObject {
+  readonly type: string;
+  readonly name: string;
+  readonly sql: string;
+}
+
+interface Copy {
+  readonly schema: readonly SchemaObject[];
+  readonly rows: Readonly<Record<string, readonly Record<string, SqlStorageValue>[]>>;
+}
+
+// Everything the app keeps in the object's SQLite database; workerd's own tables stay.
+const schemaOf = (sql: SqlStorage): SchemaObject[] =>
+  sql
+    .exec<{ type: string; name: string; sql: string | null }>(
+      "SELECT type, name, sql FROM sqlite_master WHERE name NOT LIKE 'sqlite_%' AND name NOT LIKE '_cf_%'",
+    )
+    .toArray()
+    .flatMap(({ type, name, sql: text }) => (text === null ? [] : [{ type, name, sql: text }]));
+
+const tablesOf = (schema: readonly SchemaObject[]): readonly string[] =>
+  schema.filter(({ type }) => type === "table").map(({ name }) => name);
+
+const copyOf = (sql: SqlStorage): Copy => {
+  const schema = schemaOf(sql);
+  return {
+    schema,
+    rows: Object.fromEntries(
+      tablesOf(schema).map((table) => [table, sql.exec(`SELECT * FROM "${table}"`).toArray()]),
+    ),
+  };
+};
+
+// Workerd keeps no point-in-time history of an object's storage, which Cloudflare restores to a
+// bookmark and then restarts the object: the copy is every table, index and trigger with its rows,
+// and restoring puts back exactly those, in one transaction.
+const restoreFrom = (storage: DurableObjectStorage, copy: Copy): void => {
+  storage.transactionSync(() => {
+    for (const table of tablesOf(schemaOf(storage.sql))) storage.sql.exec(`DROP TABLE "${table}"`);
+    for (const { sql: text } of copy.schema.filter(({ type }) => type === "table")) {
+      storage.sql.exec(text);
+    }
+    for (const [table, rows] of Object.entries(copy.rows)) {
+      for (const row of rows) {
+        const columns = Object.keys(row);
+        storage.sql.exec(
+          `INSERT INTO "${table}" (${columns.map((column) => `"${column}"`).join(", ")}) VALUES (${columns.map(() => "?").join(", ")})`,
+          ...columns.map((column) => row[column] ?? null),
+        );
+      }
+    }
+    // After the rows, so no trigger writes what the copy already holds.
+    for (const { sql: text } of copy.schema.filter(({ type }) => type !== "table")) {
+      storage.sql.exec(text);
+    }
+  });
+};
+
+let objects = 0;
+
+describe("restoring a Durable Object's storage", () => {
+  restoreContract({
+    hot: false,
+    create: async () => {
+      objects += 1;
+      const stub = env.BARE.get(env.BARE.newUniqueId());
+      const adapter = insideObject(
+        stub,
+        await runInDurableObject(stub, (_instance, state) =>
+          durableObjectAdapter({ storage: state.storage, options: {} }),
+        ),
+      );
+      let copy: Copy | null = null;
+      return {
+        storage: adapter,
+        // KV, as on Cloudflare: a restore of the object leaves it alone.
+        forgetLedger: defineForgetLedger(() =>
+          createKvForgetLedger({ namespace: env.FORGET_LEDGER, prefix: `restore-${objects}/` }),
+        ),
+        backUp: async () => {
+          copy = await runInDurableObject(stub, (_instance, state) => copyOf(state.storage.sql));
+        },
+        restore: async () => {
+          const taken = copy;
+          if (taken === null) throw new Error("no copy was taken");
+          await runInDurableObject(stub, (_instance, state) => restoreFrom(state.storage, taken));
+          return adapter;
+        },
+      };
+    },
+  });
+});