feat: share a lease among the background jobs, start them over after a restore, and report their status

.changeset/personal-data-operations.md

@@ -0,0 +1,16 @@
+---
+"@bounda-dev/core": minor
+"@bounda-dev/cli": minor
+"@bounda-dev/postgresql": minor
+"@bounda-dev/cloudflare": minor
+"@bounda-dev/react-router": minor
+---
+
+Operate the runtime's background work as one: share a lease, start over after a restore, and say where it stands.
+
+- The backfill, the key rotation, the retention and the purge of resolved dead letters each run in one instance at a time, under a lease kept in `bounda_checkpoints`; an instance that stalls past it stops at its next item.
+- A restored database is found and the background work starts over from it: the forget ledger is reconciled at once, the jobs forget what they knew, and the first instance clears the holds the copy brought back. Under the running app every instance finds it on its next pass, the `web` role through a watch of its own; restored while the app was stopped, the forget ledger's mark tells it at start, and without a ledger `bounda restored` (`app.restored()`) declares it. The generation behind it moves only after activity. `ForgetLedger.readMark`/`raiseMark` (forget-ledger migration 2), `PersonStore.releaseHolds`/`countIdle` and `PersonalKeyStore.releaseHolds`/`countWrappedUnder` are new (breaking for custom adapters and ledgers).
+- Discarding a process dead letter ends its instance with `ProcessAbandoned` in the same transaction; `ProcessStatus` gains `"abandoned"`, and the purge of resolved letters no longer decides whether an instance was given up.
+- Each subscriber's last failure is kept in `bounda_subscriber_failures` (storage migration 2), its text sealed under the person the event names whatever `runtime.telemetry.errorText` says. `app.subscriberFailures.list()`/`.get()` open it and are recorded in the access log (`subscriberFailures.list`, `subscriberFailures.get`); the export includes it as `subscriberFailures`. `Storage.subscriberFailures` is new (breaking for custom adapters), and `Delivery` failures carry the event and the error.
+- `app.status()`, also on `connect` clients, reports the tables' versions by database, the restores found, every subscriber's lag and last failure without its text, and the backfill, ledger, rotation, retention and purge. `Adapter.schemaVersions()` and `Adapter.migrate(scopes)` are new (breaking for custom adapters), and `migrateTables({ config })` migrates every database a configuration names.
+- `bounda status` prints it, with `--json`, and `--error-text` the failures' text through the recorded call; `bounda migrate` migrates the tables before a deploy, reading only the configuration; `bounda restored` declares a restore.

docs/src/content/docs/adapters/cloudflare.md

@@ -204,8 +204,10 @@ export default {
 
 `connect(stub)` gives a `BoundaClient`, the type `@bounda-dev/core` gives to what a request sees
 of an app wherever it runs: the same `commands` and `queries` as `app.commands` and
-`app.queries`, typed from your modules, plus `getLag()`, `events`, `deadLetters`,
-`rebuildReadModel`, `forget`, `exportPersonalData`, `restrict` and `unrestrict`.
+`app.queries`, typed from your modules, plus `getLag()`, `status()`, `events`, `deadLetters`,
+`subscriberFailures`, `rebuildReadModel`, `forget`, `exportPersonalData`, `restrict` and
+`unrestrict`. `bounda status` runs where Node can open the store, so a Durable Object's is read
+through `status()`, from an admin route of the Worker, say.
 Its commands resolve once the read models reflect them; `connect(stub, { consistency: "eventual" })`
 resolves them once their events are stored, for a request that reads nothing after it. A refusal comes
 back as an `Error` with the same `name`, `message`, `code` and, for validation, `issues`, whatever

docs/src/content/docs/adapters/index.mdx

@@ -86,14 +86,15 @@ it brings them up to the version the installed Bounda expects
 | Table | Holds |
 | --- | --- |
 | `bounda_events` | Every event, with its stream version, a global position and the person it names, if any, which [`exportPersonalData`](/guides/personal-data/#exporting-a-persons-data) reads by |
-| `bounda_checkpoints` | How far each projection, the policy runner, the process runner, the retention and the [forget ledger](/guides/personal-data/#backups)'s copy have read, where a paused rebuild stands, the time of the last ledger entry the database applied, when the last reconciliation ran and the wait of one that failed, and when the [key rotation](/guides/personal-data/#rotating-the-master-key) was last found complete for a set of master keys |
+| `bounda_checkpoints` | How far each projection, the policy runner, the process runner, the retention and the [forget ledger](/guides/personal-data/#backups)'s copy have read, where a paused rebuild stands, the time of the last ledger entry the database applied, when the last reconciliation ran and the wait of one that failed, when the [key rotation](/guides/personal-data/#rotating-the-master-key) was last found complete for a set of master keys, the leases of the [background jobs](/guides/deployment/#more-than-one-instance), and the counters that tell a [restore](/guides/personal-data/#restores) |
 | `bounda_inbox` | Which handler already completed for which event, so a retry does not run it again |
 | `bounda_scheduled_commands` | Scheduled commands, process deadlines and timeouts, and delayed policy runs |
 | `bounda_dead_letters` | Handler runs that gave up, with the error and the attempt count |
 | `bounda_idempotency_keys` | What a command dispatched with an `idempotencyKey` answered, until the key expires |
 | `bounda_personal_keys` | One key per person, which their [personal fields](/guides/personal-data/) are encrypted with, wrapped under the [master key](/guides/personal-data/#a-master-key) whose id it keeps, and the key's fingerprint, and the wait of a rewrap that failed; `forget` deletes it |
 | `bounda_persons` | One row per person the runtime knows something about: since when their [processing is restricted](/guides/personal-data/#restricting-processing), `NULL` while it is not, and, with a [retention](/guides/personal-data/#retention), when they were last active and last forgotten, the position of the last event recorded for them, and the wait of a forget the retention could not do |
 | `bounda_snapshots` | The last snapshot of every stream of an aggregate that takes them, with the fingerprint it is valid for |
+| `bounda_subscriber_failures` | The last failure of each subscriber, with its text encrypted under the person the event names, and whether the subscriber got past it ([observability](/reference/observability/#error-text)) |
 | `bounda_migrations` | The version of each set of tables Bounda versions, `storage`, `checkpoints` and `forget-ledger`, one row per migration with when it ran; a forget ledger or read models in a database of their own keep theirs there |
 
 Read models get one table each, named after the read model in snake_case behind `rm_`, so no

docs/src/content/docs/concepts/when-a-process-fails.md

@@ -52,7 +52,8 @@ never changes, so the instance is a stable sequence, and parking can follow it.
 So when a step fails for good, the instance parks every later event in its own stream, in the
 order it arrived, and the process runner moves on to the other instances. Retrying the dead letter
 runs the failed handler again and drains what is parked, in order, before the instance resumes;
-discarding it gives the instance up. [A failed process](/guides/dead-letters/#a-failed-process) has
+discarding it gives the instance up with `ProcessAbandoned`, which ends it as `abandoned`, a status
+of its own, so the purge of resolved letters never decides what happens to it. [A failed process](/guides/dead-letters/#a-failed-process) has
 the rules: what is parked, how deadlines interleave, and what a retry cut short does.
 
 ## Why not park them as dead letters

docs/src/content/docs/guides/dead-letters.md

@@ -76,8 +76,10 @@ export default defineConfig({
 ```
 
 A host without a polling worker, such as a Durable Object, is woken for it through `nextDueAt()`.
-A process instance whose letter was discarded stays given up once the letter is gone: the events
-that reach it are still dropped.
+One instance purges at a time, under a lease the others leave alone until it lapses. A process
+instance whose letter was discarded stays given up once the letter is gone: discarding wrote
+`ProcessAbandoned` in its stream, so the events that reach it afterwards are dropped whatever
+became of the letter.
 
 ## What a letter holds
 
@@ -139,8 +141,11 @@ process completes it, and what is parked after it is dropped, as for any complet
 
 `bounda dead-letters list` says how many events wait behind a failure (`3 events are parked behind
 it`), and so does `parked` on the letters of `app.deadLetters`. Discarding the letter gives the
-instance up: it stays failed, its parked events never run, though they stay in its history, and
-the events that reach it afterwards are dropped, as for an instance that has ended.
+instance up: in the same transaction as the discard, `ProcessAbandoned` (with the `letterId`, acted
+by whoever discards) ends it with the status `abandoned`. Its parked events never run, though they
+stay in its history, and the events that reach it afterwards are dropped, as for an instance that
+has ended. Only a discard does that: an instance whose letter is gone for any other reason keeps
+parking until its failure is retried.
 
 The failure is state as well: `ProcessFailed` names its dead letter (`letterId`), and the two are
 written in the same transaction, with the claim of the event that failed. A failure that could

docs/src/content/docs/guides/deployment.mdx

@@ -39,7 +39,7 @@ everything — the right answer until it is not.
 | Role | What `start()` does |
 | --- | --- |
 | `all` (default) | runs the dispatcher and the scheduled command worker |
-| `web` | nothing |
+| `web` | watches for a [restore](/guides/personal-data/#restores) of the database, nothing else |
 | `worker` | runs the dispatcher and the scheduled command worker |
 
 Serving HTTP is your app's business, not Bounda's: the role only decides whether this process also
@@ -92,6 +92,12 @@ instances coordinate through the database.
   read model.
 - Appends take one lock per store, which bounds writes to what one connection can commit
   ([the ceiling](/concepts/how-it-runs/#the-ceiling-with-numbers)).
+- The background jobs on personal data and dead letters, the backfill of what was written before
+  its fields were marked, the rotation of the master key, the retention and the purge of resolved
+  letters, each run in one instance at a time, under a lease in `bounda_checkpoints` that the others
+  leave alone until it lapses, two minutes after the holder last renewed it, and that an instance
+  stopping gives back. An instance that stalls past its lease stops at its next item, and the one
+  that took the lease over goes on from what the store holds.
 
 [At least once, and exactly once per batch](/concepts/delivery-guarantees/) says what each
 promise covers, and [How Bounda runs](/concepts/how-it-runs/) why, and what the ceiling of one
@@ -173,7 +179,11 @@ which may quote what the projection read, is in `errorMessage` only with
 process saw: another instance that never tried that read model reports no
 failure while its lag still grows. Alert on the lag, which every instance reads from the
 database; read `failing`, and the `subscriber failed` log line that names the event, to find out
-why. The event itself, payload included, is one read of the global stream away:
+why. The last failure of each subscriber, by any instance, is also kept in the database, with its
+text encrypted under the person the event names: `bounda status` shows it without the text, and
+`app.subscriberFailures.get("projection:orderSummary")` opens it, a read the
+[access log](/guides/personal-data/#who-opened-what) records. The event itself, payload included,
+is one read of the global stream away:
 `app.events.read({ after: 41, limit: 1 })` ([Reading the global stream](/guides/reading-the-stream/)).
 
 `waitingDeadlines`, next to `subscribers`, counts the process deadlines this instance's worker
@@ -185,9 +195,10 @@ while a runner is behind, and each runs anyway after ten rounds.
 
 ## Schema
 
-There are no migration files to write or run, and no migration step in your deploy. Bounda
-versions its own tables ([what an adapter creates](/adapters/#what-an-adapter-creates)) and brings
-them forward when the app starts, before it serves anything:
+There are no migration files to write or run. Bounda versions its own tables
+([what an adapter creates](/adapters/#what-an-adapter-creates)) and brings them forward when the
+app starts, before it serves anything, or before the deploy with
+[`bounda migrate`](/reference/cli/#bounda-migrate):
 
 - A database at the version the installed Bounda expects costs one read, and no lock.
 - One that is behind is migrated in place, one migration at a time, each in a transaction that
@@ -207,8 +218,20 @@ them forward when the app starts, before it serves anything:
 Stop every instance of the previous version before the first one of the new version starts:
 instances already running never read the version again, and their code was written for the tables
 before the migration, so a rolling deploy across a release that migrates would have them write rows
-the new code misreads. One of them that starts after the migration is refused. Each table prefix
-keeps its own versions, and each Durable Object, so each tenant, migrates its tables the first time
+the new code misreads. One of them that starts after the migration is refused. `bounda status`
+says, by database, at which version each scope of the tables is and which version this release
+migrates it to; `bounda migrate` runs the migrations of every database the configuration names, as
+a step of its own, so a large migration happens while nothing waits on it, and the new instances
+start on tables already migrated:
+
+```sh
+bounda status            # "storage  storage  1 of 2  (migrations pending: bounda migrate)"
+# stop the previous version
+bounda migrate
+# start the new one
+```
+
+Each table prefix keeps its own versions, and each Durable Object, so each tenant, migrates its tables the first time
 it starts on the new code. A [forget ledger](/guides/personal-data/#backups) in another database is
 migrated there the first time the app uses it.
 
@@ -336,9 +359,11 @@ Back up the database as the engine recommends: a copy of the SQLite file taken o
 key, wrapped under the [master key](/guides/personal-data/#a-master-key): keep the master key for
 as long as the backups. Restoring one taken before someone was forgotten brings them back, unless
 the app has a [forget ledger](/guides/personal-data/#backups) in another database, which forgets
-them again when the app starts on the restored copy, or within `reconcileEvery` when the copy is
-restored under the running app; restart it after restoring to reconcile at once. Set the ledger's
-`keepFor` longer than the oldest backup you keep.
+them again when the app starts on the restored copy, or at once when the copy is restored under the
+running app: every running instance finds that the database went back, and its background jobs
+start over (see [restores](/guides/personal-data/#restores)). Restored while the app was stopped,
+the ledger tells it at the next start, and without a ledger run `bounda restored`. Set the
+ledger's `keepFor` longer than the oldest backup you keep.
 
 ## Observability