Repository navigation
feat: share a lease among the background jobs, start them over after a restore, and report their status - #260
Merged
Merged
Conversation
benatespina
force-pushed
the
feat/personal-data-operations
branch
from
November 9, 2026 20:31
cda6f4c to
a575eb1
Compare
…over after a restore
…e's status and schema versions
…nd abandoned processes
…andoned processes
…e object's status
… the web role, and move the generation only after activity
…, and after work that writes no event
…writes, and take a forgetting the database lost for a restore
benatespina
force-pushed
the
feat/personal-data-operations
branch
from
November 9, 2026 21:49
e9a4632 to
567fddc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Operating the runtime's background work, for personal data and dead letters, as one:
bounda_checkpoints(kernel/shared/lease.ts, extracted from the backfill's). The lease is taken again before each item, so an instance that stalled past it stops at the next one and the instance that took it over goes on. Instances that do not hold it read the lease row, not the work.worker.stop()gives every lease back.runtime:generationforward at most every five seconds and remembers what it read. The first instance to find it went back counts the restore inruntime:restores, which every other instance then sees change (kernel/shared/restores.ts). On a restore the forget ledger is reconciled in the same pass, every job forgets what it knew (backfill caught up, rotation complete, pauses, the dispatcher's backoff), and the first instance clears the holds the copy brought back (PersonStore.releaseHolds,PersonalKeyStore.releaseHolds, the backfill's and the ledger's rows). Handler attempts stay: those runs happened.webrole, and no writes at rest.ForgetLedger.readMark/raiseMarkare new; marks only go up. They live in memory, in SQL (forget-ledger migration 2,<prefix>forget_ledger_marks) and in KV. They keep the generation outside the database: the mark is raised with every move of the generation (at most every five seconds, after activity), and on a stop after any write. A forgetting the ledger holds that the database does not also counts as a restore, whatever the generation says. A start that finds the database's generation below the mark has found a copy restored while the app was stopped, and starts the jobs over before serving. Without a ledger,bounda restored(app.restored()) declares the restore. An app in thewebrole runs only a restore watch, which reconciles the ledger at once on a find. The generation moves only after activity, a write to the stream or a worker pass that did work, so an idle instance writes nothing. The guide's "Restores" table says what each path guarantees.storageSchemaContract; nothing to add.ProcessAbandoned. Discarding a process dead letter ends its instance withProcessAbandoned { letterId }in the same transaction, acted by whoever discards.ProcessStatusgains"abandoned". Parking no longer reads the letter, so the purge of resolved letters never decides whether an instance was given up. An instance discarded before this release is abandoned on the next event that reaches it.bounda_subscriber_failuresis storage migration 2 on SQLite, libSQL, the Durable Object and PostgreSQL. It keeps one row per subscriber, with the text sealed under the person the event names whateverruntime.telemetry.errorTextsays.client.subscriberFailures.list()/get()open it and are recorded in the access log (subscriberFailures.list,subscriberFailures.get).connectand React Router's Cloudflare clients expose it too, and the export includes it (art. 15).bounda statusandbounda migrate.client.status()reports, by database, each table scope's version and the latest this release migrates to, the restores found, every subscriber's lag and last failure without its text, and the backfill, ledger, rotation, retention and purge.Adapter.schemaVersions(),Adapter.migrate(scopes)andmigrateTables({ config })are new.bounda status(--json,--error-textthrough the recorded call) prints it.bounda migratemigrates every database the configuration names before a deploy, reading only the configuration.subscriber_failuresis an inspected store (InspectedStore), soexpectForgotten(#253) searches it, with its own advice.Breaking for custom adapters and ledgers (
minor):Storage.subscriberFailures,Adapter.schemaVersions/migrate,ForgetLedger.readMark/raiseMark, and the new person and key store methods.Tests
bounda_*table is copied, then deleted and inserted back. It covers the rotation starting over without the copy's holds, the ledger forgetting again at once, and two instances, one of them first.ProcessAbandoned: fold, discard with actor, a discard a retry settled, a letter gone without a discard, and instances discarded before the release.storageSchemaContractcoversschemaVersions/migrateon every adapter. The app tests cover failures sealed, opened, forgotten and exported on memory and SQLite. The CLI tests coverstatusandmigrate, andtest-typescovers the new types.Checks
pnpm checkwith Docker: lint, build, generate, package and example typecheck, and tests are green after rebasing on feat(core): expectForgotten checks nothing of a forgotten person is left #253 (3518 passed; 4 timeouts under load in the generator and the Vite plugin, untouched here, pass alone).astro checkofdocsfails locally withthis.bridge.deleteEnv is not a function, the same withmain's docs (follow-up W1)./code-reviewhigh, twice: the second, on the restore additions, found a missing JSDoc (fixed) and a race on KV's mark that the next raise corrects (kept, documented). The first: 10 found. 8 fixed, 1 not a defect: sealing under the store'spersonwould lose the text of a person without a key. 1 kept: the restore watch's write every five seconds, which detection needs, listed as a follow-up.Left out
createRestoreWatch(a witness of the newest event, checks before each dispatcher pass, projections caught up on a find), as agreed with that session.