diff --git a/.changeset/forget-intention-categories.md b/.changeset/forget-intention-categories.md new file mode 100644 index 00000000..c25a230c --- /dev/null +++ b/.changeset/forget-intention-categories.md @@ -0,0 +1,17 @@ +--- +"@bounda-dev/core": minor +"@bounda-dev/cli": minor +"@bounda-dev/sqlite": minor +"@bounda-dev/postgresql": minor +"@bounda-dev/cloudflare": minor +--- + +The forget ledger's intention carries the categories a forgetting asks for, and the reconciliation commits again only those, never the person as a whole. + +- The intention names the keys in force its scope covers, every one for `forget(person)`, those of the categories named for `forget(person, { categories })`, and the key a category's retention ends; a forgetting finds a key given after it in that scope and takes a new intention, and one committed again keeps such a key. +- `PersonalDataForgotten.fingerprints` names every key the forgetting covered, destroyed or retained. +- A forgetting of a category's retention is committed again only while that category's retention would still end its keys, and destroys only those. +- `bounda status` and `app.status().retention` give a line for each retention: the app's, which forgets persons as a whole, and each declared category's, with how many are past it and when the next comes due, and how many keys a forgetting kept that no legal obligation keeps any more (`RetentionStatus` becomes `{ categories, keptBeyond }`, each a `CategoryRetentionStatus`). It is `null` only when no schema marks personal data. +- Breaking for adapters: `PersonalKeyStore` gains `countDue`, `countRetained` and `releaseDueHolds`; a restore clears the holds of the keys past their retention too. The storage migration that keys each category is now version 3, after the subscriber failures, and the ledger's categories column version 4, after its reason and keys. +- The KV forget ledger keeps in the key's value the categories and keys of an entry too large for KV's metadata, and reads them, at once, for the entries it hands out alone; `planKvForgetLedgerExpiry` plans such a key with `value: null`, which `expireKvForgetLedger` and `bounda forget-ledger expire` read before writing it again, one at a time, leaving alone a key gone meanwhile. +- `PersonalDataForgotten.whole` is `false`, and the category is not among `categories`, when a forgetting committed again keeps a key given to the person after it. diff --git a/.changeset/personal-data-categories.md b/.changeset/personal-data-categories.md new file mode 100644 index 00000000..1ee89bfe --- /dev/null +++ b/.changeset/personal-data-categories.md @@ -0,0 +1,16 @@ +--- +"@bounda-dev/core": minor +"@bounda-dev/cli": minor +"@bounda-dev/sqlite": minor +"@bounda-dev/postgresql": minor +"@bounda-dev/cloudflare": minor +--- + +Categories of personal data, each with its own key. `app/personal-data.ts` declares the categories with their legal basis (article 6 of the GDPR) and their retention, and `personal(schema, { category })` and `.personal({ category })` put a field in one; a field marked without one is in the default category, which behaves as every personal field did before. A misspelled category does not compile, and the app refuses to start on one. + +- A person has a key per category. `forget(person)` forgets every category but those whose basis is `legal-obligation` while their retention runs: those are kept, restricted, and forgotten on their own when it ends. `forget(person, { categories })` forgets those alone, as when a consent is withdrawn; policies and processes take the same option. +- Each declared category is forgotten after its own retention, counted from the person's last activity; `runtime.personalData.retention` is the default category's, and forgets the person as a whole. Retentions take whole calendar months and years, `"18mo"`, `"6y"`, counted on the calendar, besides durations (`RetentionInput`); `ResolvedConfig.runtime.personalData.retentionMs` becomes `retention`. +- A field moved to another category is sealed anew under its new category's key by the backfill, everywhere it is stored; going back to earlier marks walks the stream again. What a forgetting kept is destroyed once its category is no longer a legal obligation's. +- `PersonalDataForgotten` gains `whole`, `categories`, `retained` and `expired`. The export names each field's category and every category's basis, retention and, for one kept, until when. +- Breaking for adapters: `PersonalKeyStore` keys by person and category (`get(person, category)`, `find`, `list`, `add(person, category, key)`, `remove(person, fingerprint)`, `retain`, `touch`, `listDue`, `holdDue`, `nextDueHoldEnd`, `listRetained`, `nextRetainedHoldEnd`), the snapshot, idempotency, scheduler and dead-letter stores' `forget` take the fingerprints to keep, `checkAppend` takes `keyOf`, `LoadResult.fingerprints` becomes `opened`, `StoredPerson` gains `position`, and forget ledger entries gain `categories`. The SQLite, PostgreSQL and Durable Object stores migrate their tables forward on start. +- CLI: `bounda forget [--category …]` and `bounda export `; `bounda forgotten` lines carry `categories`. diff --git a/docs/src/content/docs/adapters/cloudflare.md b/docs/src/content/docs/adapters/cloudflare.md index 8fdb15cc..3d70ef79 100644 --- a/docs/src/content/docs/adapters/cloudflare.md +++ b/docs/src/content/docs/adapters/cloudflare.md @@ -169,7 +169,10 @@ reconciliation never wakes an object on its own, and with `keepFor` each key exp itself; `bounda forget-ledger expire` gives the keys written before `keepFor` was set theirs. Every forgetting writes its intention to KV before the object commits it, so one the object did not get to commit, because it was evicted or its transaction failed, is committed at its next start; -with `strict`, a forgetting KV does not take fails and forgets nobody. See +with `strict`, a forgetting KV does not take fails and forgets nobody. An entry lives in its key's +metadata, which KV caps at 1024 bytes; the categories and keys of a forgetting with too many to fit +there, some thirty, go to the key's value, which a read fetches for the entries it hands out alone, +and `bounda forget-ledger expire` writes back as it was, leaving alone a key gone meanwhile. See [backups](/guides/personal-data/#backups). ## One object per tenant diff --git a/docs/src/content/docs/guides/personal-data.md b/docs/src/content/docs/guides/personal-data.md index 3a090462..5e352837 100644 --- a/docs/src/content/docs/guides/personal-data.md +++ b/docs/src/content/docs/guides/personal-data.md @@ -1,12 +1,13 @@ --- title: Personal data -description: Mark the fields of an event, a process state or a read model that hold personal data, say whose they are, seal what was written before a field was marked, forget a person or export their data with one call, record who opened whose data, forget them after a retention or when a purpose ends, wrap their keys under a master key kept out of the database, handle forgetting in every read model that showed them, and check in a test that nothing of a forgotten person is left readable. +description: Mark the fields of an event, a process state or a read model that hold personal data, say whose they are, sort them in categories with their legal basis and retention, seal what was written before a field was marked, forget a person or export their data with one call, record who opened whose data, forget them after a retention or when a purpose ends, wrap their keys under a master key kept out of the database, handle forgetting in every read model that showed them, and check in a test that nothing of a forgotten person is left readable. sidebar: order: 11 --- Events are never deleted, so personal data in one is forgotten by making it unreadable: each -person has a key, their fields are stored encrypted with it, and `forget` throws the key away. +person has a key for each [category](#categories) of their data, their fields are stored encrypted +with the key of the field's category, and `forget` throws the keys away. [Forgetting in an immutable log](/concepts/forgetting-in-an-immutable-log/) says when to do this and when to keep the data behind a port instead. This page is the how. @@ -47,6 +48,92 @@ export const payload = ({ z, personal, person }: Event.PayloadArgs) => An event names at most one person. +## Categories + +Not all of a person's data goes at the same time. A profile is kept while they are a customer; an +invoice, for the years the law asks, even if they ask to be forgotten; a newsletter address, until +they withdraw their consent. Each kind is a category, declared in `app/personal-data.ts` with its +legal basis, as article 6 of the GDPR names it, and its retention: + +```ts +// app/personal-data.ts +import type { PersonalData } from "./+types/personal-data"; + +export const categories = { + profile: { basis: "contract", retention: "30d" }, + marketing: { basis: "consent", retention: "730d" }, + invoices: { basis: "legal-obligation", retention: "6y" }, +} satisfies PersonalData.Categories; +``` + +The basis is one of `consent`, `contract`, `legal-obligation`, `vital-interests`, `public-task` and +`legitimate-interests`. The retention is how long the category is kept after the person's last +activity: a duration as the configuration writes them, `"30d"`, or whole calendar months or +years, `"18mo"`, `"6y"`, which a legal period is given in. A calendar period ends on the same day of +the month and at the same time, leap years and months of unequal length counted as they fall; a day +the end's month lacks, as the 29th of February six years on, ends at the very end of that month's +last day, so a period is never cut short. `runtime.personalData.retention` takes the same. A field takes a category with `personal`'s second argument, and a read +model's field with `.personal`'s: + +```ts +export const payload = ({ z, personal, person }: Event.PayloadArgs) => + z.object({ + customerId: person(z.string()), + total: z.number(), + billingAddress: personal(z.string(), { category: "invoices" }), + deliveryNote: personal(z.string(), { category: "profile" }).optional(), + }); +``` + +The generator registers the names, so a category the file does not declare, in `personal`, +`.personal` or `forget`, does not compile, and `.bounda/types.ts` fails to compile while the file +declares a category `default` or one without a basis Bounda knows or a retention; the app refuses +to start on the same, for a project that skipped the type check. A field marked without a +category is in the default category, `default`, whose retention is +[`runtime.personalData.retention`](#inactivity) and which declares no basis: an app without +`app/personal-data.ts` has that category alone, and behaves as if categories did not exist. + +Each person has a key per category, created when a field of that category is first sealed for +them, so an event or a command that holds fields of several categories seals each under its own +key, and each category is forgotten on its own: + +- `forget(person)` forgets every category, as below. +- `forget(person, { categories: ["marketing"] })` forgets those alone, as when a consent is + withdrawn: their fields read as `null`, the rest stays readable, and the person is not forgotten + as a whole. +- Each category is forgotten after its own [retention](#each-category-for-its-own-time). + +A category whose basis is `legal-obligation` outlives a `forget` while its retention, counted from +the person's last activity, has not run out: the law that requires the data outweighs the request +to erase it (article 17(3)(b)). Its key is kept, restricted, and no longer in force: what it sealed +still opens, for the export and for the reads the law needs, but nothing new is sealed under it, +no snapshot or remembered rejection holds its data, and the scheduled commands and dead letters +that held it go with the rest. The key's last activity is fixed at the forgetting, so a person who +comes back extends nothing kept from before: what they write afterwards gets a new key in force, +and the kept one goes on its own once its retention has passed, with a `PersonalDataForgotten` +that names the category in `expired`. Whether a key is kept that way, and since when, is recorded +on its row. A category whose retention had already run out at the forgetting is forgotten with the +rest. + +### Changing a category + +A category's retention can change at any time: the sweep reads it as it runs, so a longer or a +shorter one applies at once to every person, what a forgetting kept included. So does a basis, and +a category removed from the file: a key a forgetting kept under a legal obligation is destroyed by +the next pass of the worker once its category is no longer a `legal-obligation` one, with a +`PersonalDataForgotten` that names it in `expired`, since nothing allows keeping it any more. + +A field moved to another category, or out of a category into the default one or back, is sealed +anew under the person's key of its new category by the [backfill](#marking-a-field-later), in every +event, process state, scheduled command and dead letter that holds it: the marks changed, so the +worker walks the stream again, opens each value sealed under a category its field is no longer in +and seals it under the right one, as the forgettings since it was written say, a category kept or +forgotten included. From then on forgetting the old category no longer reaches the value, and +forgetting the new one does. A remembered rejection that may quote it under the old key is dropped, +as is a snapshot of its stream, and decided or folded again. A value no key opens any more, its +person forgotten in its old category, stays as it is: it was forgotten already. Going back to +earlier marks walks the stream again too. + ## In a command's payload A command is stored too when it waits: dispatched with `delay`, retried by the scheduler after a @@ -198,22 +285,40 @@ await app.catchUpReadModels({ through: forgotten }); `forget` is on every `BoundaClient`: the app, the Durable Object through `connect`, the `bounda` context of React Router. A client bound with `as` records who asked on `PersonalDataForgotten`, in -`metadata.actor`, and so on `ProcessingRestricted` and `ProcessingRestrictionLifted`. In one transaction it deletes the person's key, cancels the scheduled -commands that hold their data, removes the dead letters of those that gave up, and appends -`PersonalDataForgotten` to the stream `bounda:`: the runtime's own aggregate, `bounda`, -has one stream per person it forgot. Its payload is `{ personId, reason, fingerprints, -cancelledCommands, removedDeadLetters }`: `reason` is `"requested"` for `forget` and -`"retention"` when the [retention](#retention) forgot them, `fingerprints` names the key it -destroyed, by its check value, which opens nothing, none for a person who held none, and the rest -are the ids and types of what went, the only record of commands that never became events. A -forgetting a [forget ledger](#backups) commits again keeps its id, its time and its reason. It resolves like a command, with the event's +`metadata.actor`, and so on `ProcessingRestricted` and `ProcessingRestrictionLifted`. In one transaction it deletes the person's keys, but those a +[legal obligation keeps](#categories), cancels the scheduled commands that hold their data, removes +the dead letters of those that gave up, and appends `PersonalDataForgotten` to the stream +`bounda:`: the runtime's own aggregate, `bounda`, has one stream per person it forgot. +Its payload is `{ personId, reason, whole, categories, retained, expired, fingerprints, +cancelledCommands, removedDeadLetters }`: + +- `reason` is `"requested"` for `forget` and `"retention"` when the [retention](#retention) forgot + them. A forgetting a [forget ledger](#backups) commits again keeps its id, its time and its + reason. +- `whole` is `true` when the person was forgotten as a whole, and `false` for + `forget(person, { categories })` or a category's retention, and for a forgetting committed again + from the [ledger](#backups) that kept a key given to the person after it: that key's category is + then not among `categories` either. +- `categories` are the categories forgotten, every one but those `retained` when `whole`; + `retained`, those a legal obligation kept; `expired`, those kept at an earlier forgetting whose + retention ended now. +- `fingerprints` names the keys it covered, destroyed or retained, by their check value, which + opens nothing; none for a person who held none. +- The rest are the ids and types of what went, the only record of commands that never became + events. + +`forget(person, { categories })` forgets those categories alone: their keys go, and so do the +scheduled commands, dead letters, snapshots and remembered rejections that hold data sealed under +them; the rest of the person's data, and what holds it, stays. Naming a category the app does not +have, or none, is refused with `ValidationError`. It resolves like a command, with the event's position, so `catchUpReadModels({ through })` waits for the read models that handle it, and the Durable Object's `forget` and the one in React Router's `bounda` context wait for them themselves unless `consistency` is `"eventual"`. A person without a key is forgotten all the same, since a read model may hold what their events carried in the clear, and forgetting twice appends twice. An empty id is refused with `ValidationError`. -From then on every personal field of theirs reads as `null`, in every event, however old. A -person registered again gets a fresh key; their old events stay unreadable. A command running +From then on every personal field of theirs reads as `null`, in every event, however old, but +those of a category kept under a legal obligation. A person registered again gets fresh keys; +their old events stay unreadable. A command running when `forget` commits stores nothing it decided on their data, nor anything sealed under the key `forget` removed: the state it loaded remembers the key each person's data was read under, its events name the key they seal each person's fields under, and the store appends them only while @@ -231,9 +336,9 @@ dropped: its person was forgotten`. The [dead letters](/guides/dead-letters/) li scheduled command's payload with its fields opened. A [snapshot](/concepts/long-streams/#snapshots) holds the state in the clear, so the same -transaction removes every snapshot whose state carried a field of theirs; each one names the -people it holds and the key it read each one under, and none is written for a person who no -longer holds that key. The next command on such an +transaction removes every snapshot whose state carried a field of theirs under a key that goes or is +kept; each one names the people it holds and the key it read each one under, and none is written +for a person who no longer holds that key in force. The next command on such an aggregate folds it from its first event, with the fields forgotten, and the snapshot after it holds them as `null`. @@ -245,10 +350,10 @@ them. What it cannot see is personal data that came in the command's payload, marked or not: keep that out of a rejection's message and say what went wrong with its code and the ids involved. -The message of an error a handler throws may quote a personal field it read, so a [dead -letter](/guides/dead-letters/#what-a-letter-holds) keeps the message and the stack encrypted with -the key of the person its work names, and `forget` leaves the letter in place with them reading as -`null`. The key is the one the work read their data under: a person forgotten while the handler +The message of an error a handler throws may quote a personal field it read, of any category, so a +[dead letter](/guides/dead-letters/#what-a-letter-holds) keeps the message and the stack encrypted +under every key the person its work names holds, one inside the other, and forgetting any category +of theirs leaves the letter in place with them reading as `null`. The keys are those the work read their data under: a person forgotten while the handler was failing, and given a fresh key before it gave up, has the text encrypted under a key nobody keeps, so it reads as `null` too instead of opening under the fresh key; the stream's `ScheduledCommandFailed` and `ProcessFailed` name the error's class and code and never its text, and so does the last error the inbox and the scheduler keep of a claim. @@ -290,28 +395,46 @@ person, for them to take away, as the right of access and the right to portabili { personId: "c-42", exportedAt: "2026-03-01T10:00:00.000Z", + categories: { + default: { basis: null, retention: "730d", restrictedUntil: null }, + profile: { basis: "contract", retention: "30d", restrictedUntil: null }, + invoices: { basis: "legal-obligation", retention: "6y", restrictedUntil: null }, + }, events: [ { id, type: "CustomerRegistered", aggregateType: "customer", aggregateId: "c-42", - version: 1, position: 17, timestamp, data: { email: "ana@example.com" } }, + version: 1, position: 17, timestamp, data: { email: "ana@example.com" }, + categories: { email: "default" } }, { id, type: "OrderPlaced", aggregateType: "order", aggregateId: "o-9", - version: 1, position: 40, timestamp, data: { deliveryNote: "ring twice" } }, + version: 1, position: 40, timestamp, + data: { billingAddress: "1 Main St", deliveryNote: "ring twice" }, + categories: { billingAddress: "invoices", deliveryNote: "profile" } }, + ], + scheduledCommands: [ + { commandId, type: "RemindCustomer", executeAt, data: { email }, categories: { email: "default" } }, ], - scheduledCommands: [{ commandId, type: "RemindCustomer", executeAt, data: { email } }], deadLetters: [ { id, kind: "policy", handler: "customer.welcomeOnCustomerRegistered", - errorMessage: "no mailbox for ana@example.com", errorStack, data: {} }, + errorMessage: "no mailbox for ana@example.com", errorStack, data: {}, categories: {} }, ], subscriberFailures: [ { subscriber: "projection:customerDirectory", eventId, eventType: "CustomerRegistered", errorMessage: "cannot index ana@example.com", errorStack }, ], readModels: { - customerDirectory: [{ key: "c-42", data: { email: "ana@example.com" } }], - orderSummary: [{ key: "o-9", data: {} }], + customerDirectory: [ + { key: "c-42", data: { email: "ana@example.com" }, categories: { email: "default" } }, + ], + orderSummary: [{ key: "o-9", data: {}, categories: {} }], }, } ``` +`categories` describes every category of the app, as article 15(1) of the GDPR asks: why it is +processed, its `basis`, `null` for the default category, which declares none; how long it is kept +after their last activity, its `retention`, `null` when nothing forgets it for inactivity; and, +for a category a legal obligation kept past a forgetting, when what was kept goes, +`restrictedUntil`. Every record lists the category of each field of its `data` in `categories`. + `events` are every event that names the person, by its field marked `person` or its aggregate's id, in position order, the events of their own stream `bounda:` included; `data` holds the fields marked `personal` of that event, opened, and nothing else of its payload, so an event @@ -320,10 +443,10 @@ names](#in-a-processs-state), and its `data` holds the personal fields of that s whose payload holds personal fields of theirs, with `commandId` the id the dispatch answered or, for one the runtime scheduled itself, the scheduler's key, each with the same `data`. `deadLetters` are the handler runs that gave up and name the person: work about them, whose -`errorMessage` and `errorStack` are encrypted with their key and come out opened, and the +`errorMessage` and `errorStack` are encrypted with their keys and come out opened, and the scheduled commands whose payload holds their fields, with its `data`. `subscriberFailures` are the last failures of the subscribers on an event that names them, kept with their error text -encrypted with their key (see [observability](/reference/observability/#error-text)), opened. +encrypted with their keys (see [observability](/reference/observability/#error-text)), opened. After `forget`, the person's events and letters are still theirs, and every field and error text reads as `null`, as a handler reads it. `readModels` has an entry for every read model with a field marked [`.person()`](#personal-data-in-read-models): the rows whose person field holds the id, each @@ -368,8 +491,9 @@ id of the person the row is about, a `string` column, at most one per read model methods of the field, next to `.index()`, because a read model's columns are declared there; a person who appears in many rows, such as a customer in an order summary, is marked on that foreign column (`customerId: f.string().index().person()`), with its own index if the export of -a person with many rows must be fast. The marks change nothing about how the adapter stores the -column. They do three things: the person's [export](#exporting-a-persons-data) includes the rows +a person with many rows must be fast. `.personal({ category: "invoices" })` puts the column in a +[category](#categories), which the export names; without it the column is in the default one. The +marks change nothing about how the adapter stores the column. They do three things: the person's [export](#exporting-a-persons-data) includes the rows that name them, a [query](#who-opened-what) of a read model with personal fields is recorded in the access log, and the generator checks forgetting by them. @@ -386,16 +510,21 @@ export const project = async ({ event, table }: Projection.Args) => { ``` What it does is the read model's business: delete the row, as here, keep it without the personal -columns by upserting it again without them, or keep a count. The generator checks it: -`.bounda/types.ts` declares one type per read model, `CustomerDirectoryForgetting`, that fails to -compile, naming the read model and what to add, while the read model marks more than one field -with `.person()`, marks fields with `.personal()` and none with `.person()`, or has fields marked -`.personal()` or projects an event with personal fields and has no projection of -`PersonalDataForgotten`. The app refuses to start on the same marks with `ConfigurationError`, -for a project that skipped the type check. A read model that projects personal events and marks -nothing is still checked for forgetting, and nothing more: Bounda cannot tell whether it kept a -copy, and one that keeps only the order's total must not be made to mark anything. A person field -alone holds an id, as events do, and needs no forgetting. +columns by upserting it again without them, or keep a count. Its payload says which +[categories](#categories) went: `categories`, every one but those a legal obligation kept when the +person was forgotten as a whole, and only those named when a consent was withdrawn (`whole: +false`). A read model clears what it holds of those categories alone, so a newsletter's subscribers, +all of them `marketing`, go when that category does, and stay when only an invoice is at stake: + +```ts +// app/read/newsletter/projections/bounda/personal-data-forgotten.ts +import type { Projection } from "./+types/personal-data-forgotten"; + +export const project = async ({ event, table }: Projection.Args) => { + const { personId, categories } = event.payload; + if (categories.includes("marketing")) await table.delete({ subscriberId: personId }); +}; +``` A projection of a personal event handles `null` as well, since a rebuild replays every event, the forgotten ones included: a `string` column that is `NOT NULL` takes a `null` email badly, so make @@ -634,35 +763,51 @@ export const handler = async ({ aggregateId, forget }: Process.DeadlineArgs) => }; ``` -Policies and processes receive `forget(personId)` in their arguments, next to `commands`. It +Policies and processes receive `forget(personId, options)` in their arguments, next to `commands`, +with the same `categories` a client's `forget` takes. It forgets as `forget` on the app does, with `reason: "requested"`, inside the run's own unit of work: the forgetting commits with the run's commands and its step, in the same transaction, or not at all, so a run that fails and is retried forgets once. It resolves at once, before the commit, and rejects an empty id with `ValidationError`. The `PersonalDataForgotten` it appends is caused by the event the handler reacted to, one hop deeper, as its commands are. -### One key per person - -Each person has one key, so their marked fields are forgotten together. "Keep the invoices six -years, the profile thirty days" does not fit one key: put the profile in marked events, which the -retention or a process forgets, and keep what the law makes you hold longer, such as an invoice, -behind a [port](/guides/calling-the-outside-world/) with a retention of its own, or in events whose -fields you do not mark. +### Each category for its own time + +"Keep the invoices six years, the profile thirty days" is two [categories](#categories), each with +its retention. A declared category is forgotten on its own once the person has been inactive for +its retention, as `forget(person, { categories })` does, with `reason: "retention"` and `whole: +false`: the profile goes thirty days after their last activity, the invoices six years after it, +whatever the other categories do. A category whose basis is `legal-obligation` is forgotten that way +too when its retention ends, even when the app's retention forgot the person as a whole earlier: +that forgetting kept it, and its time runs from the activity it kept. + +Activity is the person's, as for the app's retention: any event of the app's aggregates that names +them, whatever category it carries, so a customer who keeps ordering keeps their profile. The sweep +reads each category's keys by their last activity, recorded on every key the person holds in force +by the same subscriber, `retention`, which runs whenever the app has a retention or declares a +category; a person whose processing is restricted is left alone, and a key the sweep could not +forget waits on its own row, as a person does. `runtime.personalData.retention` is the default +category's retention, and forgets the person as a whole: every category but those a legal +obligation keeps, after which each declared category has nothing left to forget but what was +kept. ## Where the keys live Each adapter keeps the keys in a table next to the events, `bounda_personal_keys`, one row per -person with the key, the id of the key it is wrapped under and its fingerprint: the memory adapter +key with the person, the category, the key, the id of the key it is wrapped under, its +fingerprint, the person's last activity and, for one a legal obligation kept past a forgetting, +since when: at most one key in force per person and category. Each key is named by the person and +its fingerprint: the memory adapter in memory, SQLite and the Durable Object in their database, PostgreSQL in its schema. The fingerprint is the key's AES-GCM tag of nothing under a nonce no value is ever sealed under, a key check value: it names the key without revealing anything of it, and the runtime computes it from the key itself, so it checks every key it reads against it, even one it can never export. Whether a person's processing is restricted, and since when, lives apart, in `bounda_persons`, since a person without a key is restricted all the same, and so does, with a -retention, when they were last active and last forgotten. A key is created when the first event -with a personal field of that person is stored, by the transaction that stores it, and deleted by -`forget`. Encryption is AES-256-GCM through WebCrypto, with a nonce per value, on Node and in +retention, when they were last active and last forgotten as a whole. A key is created when the +first event with a personal field of that person in its category is stored, by the transaction +that stores it, and deleted by `forget`. Encryption is AES-256-GCM through WebCrypto, with a nonce per value, on Node and in `workerd` alike; what the store holds for a personal field is an envelope with the ciphertext, the -person's id and the key's fingerprint, which the [global stream](/guides/reading-the-stream/) +person's id, the category unless it is the default one, and the key's fingerprint, which the [global stream](/guides/reading-the-stream/) shows as it is; a value sealed under a key the person no longer has, because they were forgotten and registered again, reads as `null` next to what their new key opens. What is read is opened by that shape, whatever the schema marks, so an event or a delayed command whose handler or caller @@ -832,11 +977,15 @@ is restored, to apply the erasures again. A forget ledger is how Bounda does the ### The forget ledger The ledger is a record of every forgetting kept outside the database: for each `PersonalDataForgotten`, -the person's id, the event's id, its time, its reason, and the fingerprints of the keys it -destroys, and nothing else. It is a suppression list, the one thing about a forgotten person that -has to outlive their data, so it holds no field of theirs; a fingerprint is a check value of a key, -which opens nothing and names nothing once the key is gone. Give it a store a restore of the -database does not touch: +the person's id, the event's id, its time, its reason, the categories it asked for, `null` for +the person as a whole, and the fingerprints of the keys in force it covers, and nothing else. It is +a suppression list, the one thing about a forgotten person that has to outlive their data, so it +holds no field of theirs; a fingerprint is a check value of a key, which opens nothing and names +nothing once the key is gone. Forgetting someone again forgets what that forgetting asked for: a +withdrawn consent again, never the person as a whole, nor the invoices a legal obligation keeps, +and those kept again while their retention runs; a forgetting that only ended what an earlier one +kept asks for nothing, since the retention ends it again on its own. Give it a store a restore of +the database does not touch: ```ts import { defineConfig, masterKey } from "@bounda-dev/core/config"; @@ -1041,14 +1190,18 @@ bounda forget --from forgotten.jsonl ``` `bounda forgotten` prints one line of JSON per forgetting, the entry a ledger would hold, reading -the event store without starting the app; `bounda forget` commits again each forgetting listed, -under its own id, time and reason, as a reconciliation would: unless the database holds it, and -only while the person holds a key it names. With a ledger configured, it records each in the -ledger first. A line may also be a bare person's id, for a list kept some other way, which is -forgotten, as a request, whenever the person holds a key: list only those forgotten after the copy -was taken. `--actor ` names who does it on each `PersonalDataForgotten` it appends. -`app.forgetAgain(entries, { actor })` is the same in code. Then run `bounda restored`, so the -jobs start over from the copy ([restores](#restores)). +the event store without starting the app; `bounda forget --from` commits again each forgetting +listed, of the categories it asked for or of the person as a whole, under its own id, time and +reason, as a reconciliation would: unless the database holds it, and only while the person holds a +key in force it covers and names. With a ledger configured, it records each in the ledger first. A +line may also be a bare person's id, for a list kept some other way, which is forgotten as a whole, +as a request, whenever the person holds a key: list only those forgotten after the copy was taken. +`--actor ` names who does it on each `PersonalDataForgotten` it appends. +`app.forgetAgain(entries, { actor })` is the same in code, each entry with its `categories`. Then +run `bounda restored`, so the jobs start over from the copy ([restores](#restores)). Without a +backup in play, `bounda forget ` forgets a person from the command line, `--category` +naming the categories alone, and `bounda export ` prints their data: see the +[CLI](/reference/cli/#bounda-export-and-bounda-forget-person). Without either, a restored backup brings back the data of everyone forgotten after it was taken, until they are forgotten again. @@ -1128,7 +1281,7 @@ await expectForgotten(app, "c-1"); It looks for every value the person had: each value of a field marked `personal()` that the test app sealed under a key they no longer hold, in an event, a process's state, a command it stored or a dead letter, a cancelled command's included, and each value a read model kept in a field marked -`.personal()` for them before they were forgotten. It searches every place the runtime keeps +`.personal()` for them before its category was forgotten. It searches every place the runtime keeps something, read as stored, never through the APIs that open fields: every table of the storage and of each read model's adapter, unmarked columns and JSON documents included, so a projection that copied the email into a column it did not mark is found; the forget ledger; and every entry @@ -1188,9 +1341,16 @@ try { - `places`: anything else, under the name the report gives it: what a test double of a port was sent, so a test proves the [policy that tells the outside world](#telling-the-outside-world) erased it there too. - -It refuses to pass for want of something to look for: a person who still holds their key, or one -with no value to look for and no `values`, rejects with an `Error` saying so. +- `categories`: the [categories](#categories) to check alone, after + `forget(personId, { categories })`: what was sealed or kept of the person in them, and the + fields of `before` in them, are looked for, and the person must hold no key in force of them. + +Without `categories` it looks for what was sealed under every key the person no longer holds, so a +category a legal obligation kept past the forgetting, whose key they still hold, is left out: that +data stays lawfully, and its values may turn up anywhere it was kept. It refuses to pass for want +of something to look for: a person who still holds their keys, or one with no value to look for +and no `values`, rejects with an `Error` saying so, and so does one who still holds a key in force +of a category `categories` names. `expectForgotten` also takes an adapter instead of the app, and then searches that adapter's tables for `values` and `before` alone: for an app built with `createApp` or `boot()`, or a diff --git a/docs/src/content/docs/reference/cli.md b/docs/src/content/docs/reference/cli.md index 5b8704f2..99a45630 100644 --- a/docs/src/content/docs/reference/cli.md +++ b/docs/src/content/docs/reference/cli.md @@ -54,7 +54,7 @@ the same way. ## Who runs a command `--actor ` names who runs a command that acts or reads personal data: `rebuild`, -`dead-letters list`, `retry` and `discard`, `status`, and `forget`. The command acts as that id, as +`dead-letters list`, `retry` and `discard`, `status`, `export` and `forget`. The command acts as that id, as [`client.as(actor)`](/concepts/ids-causation-and-correlation/#who-acts) does in code: the events it writes carry it in their metadata, its log lines name it, and so does every access log record it causes. @@ -258,9 +258,9 @@ bounda status --error-text --actor ops-ana ``` tables - storage storage 2 of 2 + storage storage 3 of 3 storage checkpoints 1 of 1 - forgetLedger forget-ledger 1 of 1 + forgetLedger forget-ledger 4 of 4 restores found while running: 0 subscribers (head at 1204) projection:customerDirectory at 1197, 7 behind @@ -269,7 +269,10 @@ subscribers (head at 1204) personal data backfill: sealed through position 1204, caught up forget ledger: reconciled, applied through 2026-11-08T22:15:00.000Z, last run 2026-11-09T10:00:00.000Z key rotation: under 7Kx0Qw9z (previous Wn3eT1aa), 12 keys left to rewrap -retention: 730d, 0 persons past it, next due 2028-01-04T09:13:00.000Z +retention + default 2y, 0 persons past it, next due 2028-01-04T09:13:00.000Z + marketing 18mo, 3 keys past it, next due 2026-11-09T11:40:00.000Z + invoices 6y, 0 keys past it, next due 2031-02-12T16:20:00.000Z dead letter retention: 30d, next purge 2026-11-20T08:00:00.000Z ``` @@ -287,6 +290,10 @@ dead letter retention: 30d, next purge 2026-11-20T08:00:00.000Z [rotation](/guides/personal-data/#rotating-the-master-key) of the master key, the [retention](/guides/personal-data/#inactivity) and the [purge of resolved dead letters](/guides/dead-letters/#keeping-resolved-letters), each `not configured` when it is off. + The retention has a line for the app's, which forgets persons as a whole, under `default`, and + one for each category of `app/personal-data.ts`, which [forgets its + keys](/guides/personal-data/#each-category-for-its-own-time), with how many are past it now and + when the next comes due, and the keys a forgetting kept that no legal obligation keeps any more. Under the rule of [error text](#error-text) set to `"include"`, by `--error-text`, `BOUNDA_ERROR_TEXT=include` or the configuration, it also prints, under each failure, the text it @@ -332,7 +339,7 @@ bounda migrate ``` migrated - storage storage 2 of 2 + storage storage 3 of 3 storage checkpoints 1 of 1 readModels.reports checkpoints 1 of 1 ``` @@ -351,7 +358,29 @@ are refused untouched with [`STORAGE_SCHEMA_TOO_NEW`](/reference/errors/#codes). Exit codes: `0` migrated, `1` bad arguments, `2` the configuration could not be loaded or a migration failed. -## `bounda forgotten` and `bounda forget` +## `bounda export` and `bounda forget ` + +Answer a person's request from the command line. Both boot the project as `bounda dead-letters` +does, and act as `--actor`. + +```bash +bounda export c-42 --actor ops-ana > c-42.json +bounda forget c-42 --category marketing --actor ops-ana +bounda forget c-42 --actor ops-ana +``` + +`bounda export ` prints, as JSON, what `app.exportPersonalData` answers: every record that +names the person, with each field's category, and every category's basis and retention (see +[Exporting a person's data](/guides/personal-data/#exporting-a-persons-data)). It is recorded in the +access log like any export. + +`bounda forget ` forgets the person as a whole, as `app.forget` does, and prints +`forgot c-42`; with `--category`, repeated for several, only those categories, as when a consent is +withdrawn, and prints `forgot marketing of c-42`. A category whose basis is `legal-obligation` is +kept, restricted, until its retention ends, either way. A category the app does not have fails with +`VALIDATION_FAILED` and nothing is forgotten. + +## `bounda forgotten` and `bounda forget --from` Apply again, by hand, the forgettings a restored backup undid, for an app without a [forget ledger](/guides/personal-data/#backups). `bounda forget` boots the project as @@ -367,16 +396,18 @@ bounda forget --from forgotten.jsonl `bounda forgotten` reads the stream and prints, one line of JSON each, the `PersonalDataForgotten` of every person forgotten since `--since`: a date, compared with each event's time, which reads the whole stream, or a position of the stream to read after. Each line is -`{"personId":…,"eventId":…,"forgottenAt":…,"reason":…,"fingerprints":[…]}`, the entry a ledger -would hold, without `fingerprints` for a forgetting recorded before Bounda 0.4.0; the count goes to -standard error, so standard output is the list alone. Run it against the database before it is -restored, or regularly next to the backups. +`{"personId":…,"eventId":…,"forgottenAt":…,"reason":…,"fingerprints":[…],"categories":…}`, the +entry a ledger would hold: `categories` `null` for a person forgotten as a whole and else the +categories the forgetting asked for, and no `fingerprints` for a forgetting recorded before Bounda +0.4.0; the count goes to standard error, so standard output is the list alone. Run it against the +database before it is restored, or regularly next to the backups. -`bounda forget --from ` commits again each forgetting listed, under its own id, time and -reason, unless the database holds it, and only while the person holds a key the line names, as a -reconciliation would; with a forget ledger configured, it records each in the ledger first. A -line may also be a bare person's id, forgotten as a request whenever they hold a key; blank lines -are skipped. It prints each person it forgot and how many of those listed: +`bounda forget --from ` commits again each forgetting listed, of the categories it asked for +or of the person as a whole, under its own id, time and reason, unless the database holds it, and +only while the person holds a key in force the line covers and names, as a reconciliation would; +with a forget ledger configured, it records each in the ledger first. A line may also be a bare +person's id, forgotten as a whole, as a request, whenever they hold a key; blank lines are skipped. +It prints each person it forgot and how many of those listed: ``` forgot again c-42 @@ -386,12 +417,14 @@ forgot again 1 of 3 listed; the rest hold no key the line names, or this store h | Option | Applies to | Meaning | | --- | --- | --- | | `--since ` | `forgotten` | A date such as `2026-11-01` or `2026-11-01T03:00:00Z`, or a position: a whole number | -| `--from ` | `forget` | What `bounda forgotten` printed, or one person's id a line | -| `--root`, `--config`, `--registry`, `--error-text` | both | As for `bounda rebuild` | -| `--actor ` | `forget` | Who forgets them again, named in each `PersonalDataForgotten` and its log line | +| `--from ` | `forget` | What `bounda forgotten` printed, or one person's id a line; not with a person | +| `--category ` | `forget ` | Forget only this category of the person; repeat it for several | +| `--root`, `--config`, `--registry`, `--error-text` | all | As for `bounda rebuild` | +| `--actor ` | `export`, `forget` | Who acts, named in each `PersonalDataForgotten`, its log line and the access log | -Exit codes: `0` done, `1` bad arguments, `2` the project could not be loaded, or the file could not -be read or has a line that is neither a person's id nor an entry, in which case nobody is forgotten. +Exit codes: `0` done, `1` bad arguments, or a person and `--from` both or neither, `2` the project +could not be loaded, the export or the forget failed, or the file could not be read or has a line +that is neither a person's id nor an entry, in which case nobody is forgotten. ## `bounda forget-ledger expire` @@ -444,6 +477,8 @@ It throws `ConventionError` when the layout breaks a convention; `formatConventi `app.as(actor).rebuildReadModel(name)`. `bounda status` is `app.status()`, and its failures' text `app.as(actor).subscriberFailures.list()`; `bounda migrate` is `migrateTables({ config })` from `@bounda-dev/core`; `bounda restored` is `app.restored()`. `bounda dead-letters` is `app.deadLetters` on a booted app, -or `app.as(actor).deadLetters`, and `bounda forget` is `app.forgetAgain(entries, { actor })`. The +or `app.as(actor).deadLetters`, `bounda export` is `app.as(actor).exportPersonalData(person)`, +`bounda forget ` is `app.as(actor).forget(person, { categories })`, and +`bounda forget --from` is `app.forgetAgain(entries, { actor })`. The commands boot the app with `createConsoleLogger({ stream: process.stderr })` from `@bounda-dev/core/node`, which writes every level to the stream it is given. diff --git a/docs/src/content/docs/reference/configuration.md b/docs/src/content/docs/reference/configuration.md index 330ad086..505e2bea 100644 --- a/docs/src/content/docs/reference/configuration.md +++ b/docs/src/content/docs/reference/configuration.md @@ -104,9 +104,13 @@ a stopped one left: see [delivery guarantees](/concepts/delivery-guarantees/#the ### Personal data -- **`personalData.retention`**, none by default. How long a person may go without an event of the +- **`personalData.retention`**, none by default. How long, a duration or whole calendar months or + years such as `"6y"`, a person may go without an event of the app's aggregates naming them before the runtime forgets them, as `forget` does, with `reason: - "retention"`. Without it nobody is forgotten for inactivity and nothing is recorded for it. A + "retention"`, every category but those a legal obligation keeps: the retention of the default + category. Each category `app/personal-data.ts` declares has its own retention besides, see + [categories](/guides/personal-data/#categories). Without either, nobody is forgotten for + inactivity and nothing is recorded for it. A person whose processing is restricted is left alone until it is lifted; one active again after being forgotten starts a new clock. Turning it on applies it to the history: whoever was already inactive for that long is forgotten in the first passes. It is app-wide and has no override per aggregate: a person's diff --git a/docs/src/content/docs/reference/conventions.md b/docs/src/content/docs/reference/conventions.md index 3d5f1ce5..2860e380 100644 --- a/docs/src/content/docs/reference/conventions.md +++ b/docs/src/content/docs/reference/conventions.md @@ -29,6 +29,7 @@ Names are kebab-case. `` is a name you choose. | `app/domain//processes//at-.ts`, `at-timeout.ts` | `handler` | | `app/domain//processes///on-.ts` | `handler`, for that aggregate's event | | `app/audit.ts` | optional: `onPersonalDataAccess`, typed as `OnPersonalDataAccess`, where the [access log](/guides/personal-data/#who-opened-what) goes; without it, the app's logger | +| `app/personal-data.ts` | optional: `categories`, `satisfies PersonalData.Categories`, the [categories of personal data](/guides/personal-data/#categories) with their basis and retention; without it, every personal field is in the default category | | `app/read//view.ts` | `fields`; `.person()` on at most one field, `.personal()` on those that hold that person's data | | `app/read//projections//.ts` | `project`; `on` optional | | `app/read//projections/bounda/personal-data-forgotten.ts` | `project`, required once a projection of the read model sees a field marked `personal` or its view marks a field `.personal()` | @@ -75,7 +76,10 @@ field marked `personal`, or marks a field `.personal()`, and has no `projections/bounda/personal-data-forgotten.ts`; one that marks a field `.personal()` and none `.person()`; one that marks two fields `.person()`. The generated registry is checked against `Registry`, so an `app/audit.ts` whose `onPersonalDataAccess` is not an `OnPersonalDataAccess` -fails to compile there. See +fails to compile there. `PersonalDataDeclaration` fails to compile while `app/personal-data.ts` +declares a category named `default`, or one without a basis Bounda knows or a retention, and a +category that `personal({ category })`, `.personal({ category })` or `forget` names and the file +does not declare does not compile where it is written. See [Personal data in read models](/guides/personal-data/#personal-data-in-read-models). ## What boot refuses @@ -94,6 +98,10 @@ Some rules need the modules themselves, so the app checks them when it is create - a read model that marks two fields `.person()`, marks a field `.personal()` and none `.person()`, or marks a field `.personal()` and does not project `PersonalDataForgotten`; - an `app/audit.ts` that does not export `onPersonalDataAccess` as a function; +- an `app/personal-data.ts` whose `categories` is not an object, declares `default`, names a + category with anything but letters, digits, `-` and `_` after a first letter, or gives one a + basis article 6 does not name or a retention that is not a duration longer than none; a field + marked personal in a category the file does not declare; - a `payload`, `repository`, process `state` or `correlate` export that is not a function; - a configuration that names a port, an implementation, an aggregate or a read model that does not exist, or leaves out a port that has several implementations. @@ -120,9 +128,9 @@ and not failed on: | Path | Holds | | --- | --- | -| `.bounda/registry.ts` | Every module, grouped as the runtime needs it, and `app/audit.ts` as `audit` when there is one. `boot()` imports it | -| `.bounda/register.d.ts` | Registers the registry type and the type of the `ports` section with `@bounda-dev/core/register`, so `boot()` and `BoundaApp` are typed for the project without a type argument and `defineConfig` checks the implementation names | -| `.bounda/types.ts` | The state, events, ports, commands, rows and queries maps the `+types` build on, how each read model stores its fields (`FieldTypes`, which decides what its table filters each field by), and one `Forgetting` check per read model | +| `.bounda/registry.ts` | Every module, grouped as the runtime needs it, and `app/audit.ts` as `audit` and `app/personal-data.ts` as `personalData` when there are. `boot()` imports it | +| `.bounda/register.d.ts` | Registers the registry type, the type of the `ports` section and the names of the categories of personal data with `@bounda-dev/core/register`, so `boot()` and `BoundaApp` are typed for the project without a type argument, `defineConfig` checks the implementation names, and `personal({ category })` and `forget` check the categories | +| `.bounda/types.ts` | The state, events, ports, commands, rows and queries maps the `+types` build on, how each read model stores its fields (`FieldTypes`, which decides what its table filters each field by), one `Forgetting` check per read model, `PersonalCategory`, the names `app/personal-data.ts` declares, and its `PersonalDataDeclaration` check | | `**/+types/.ts` | The argument types each module imports. An implementation has none: it imports its port | They are derived from your code, so they are not versioned. `tsconfig.json` must include them as diff --git a/docs/src/content/docs/reference/limitations.md b/docs/src/content/docs/reference/limitations.md index f579a968..acfb743b 100644 --- a/docs/src/content/docs/reference/limitations.md +++ b/docs/src/content/docs/reference/limitations.md @@ -47,7 +47,9 @@ client, and nothing it does not see: What a production app does get, and where it is explained: [rebuilding a read model](/guides/deployment/#rebuilding-a-read-model) without taking it offline, -[forgetting a person](/guides/personal-data/) without deleting an event, +[forgetting a person](/guides/personal-data/) without deleting an event, each +[category](/guides/personal-data/#categories) of their data with its basis and retention, what a +legal obligation keeps kept, [exporting their data](/guides/personal-data/#exporting-a-persons-data), read models included, [recording who opened it](/guides/personal-data/#who-opened-what) and [restricting their processing](/guides/personal-data/#restricting-processing) without losing a diff --git a/packages/cli/src/forget-ledger-expire.test.ts b/packages/cli/src/forget-ledger-expire.test.ts index 59752727..8df22bfd 100644 --- a/packages/cli/src/forget-ledger-expire.test.ts +++ b/packages/cli/src/forget-ledger-expire.test.ts @@ -23,12 +23,13 @@ const metadata = (daysAgo: number) => ({ }); // Answers the listing with `listed`, and keeps what each bulk command was handed. -const fakeWrangler = (listed: unknown) => { +const fakeWrangler = (listed: unknown, values: Readonly> = {}) => { const calls: (readonly string[])[] = []; const files: unknown[] = []; const wrangler: Wrangler = async (args) => { calls.push(args); if (args[1] === "bulk") files.push(JSON.parse(await readFile(args[3] ?? "", "utf8"))); + if (args[2] === "get") return values[args[3] ?? ""] ?? ""; return args[1] === "key" ? JSON.stringify(listed) : ""; }; return { wrangler, calls, files }; @@ -90,6 +91,47 @@ describe("expiring a KV forget ledger through wrangler", () => { ]); }); + it("writes again with its value a key whose value holds the entry, and leaves alone one gone", async () => { + const valued = `${ROOT}t/1:e-1`; + const held = JSON.stringify({ categories: null, fingerprints: ["k-1"] }); + const { wrangler, calls, files } = fakeWrangler( + [ + { name: valued, metadata: { ...metadata(1), inValue: true } }, + { name: `${ROOT}t/2:e-2`, metadata: { ...metadata(1), inValue: true } }, + { name: `${ROOT}t/3:e-3`, metadata: { ...metadata(1), inValue: true } }, + ], + { [valued]: held, [`${ROOT}t/2:e-2`]: "Value not found", [`${ROOT}t/3:e-3`]: "[1]" }, + ); + await expireKvLedger({ + wrangler, + ledger, + binding: "FORGET_LEDGER", + keepForMs: 35 * DAY, + target: { local: true }, + now: NOW, + }); + expect(calls[1]).toEqual([ + "kv", + "key", + "get", + valued, + "--binding", + "FORGET_LEDGER", + "--text", + "--local", + ]); + expect(files).toEqual([ + [ + { + key: valued, + value: held, + expiration: (NOW + 34 * DAY) / 1000, + metadata: { ...metadata(1), inValue: true }, + }, + ], + ]); + }); + it("calls no bulk command when nothing changes, and names the target it was given", async () => { const { wrangler, calls } = fakeWrangler([]); expect( diff --git a/packages/cli/src/forget-ledger-expire.ts b/packages/cli/src/forget-ledger-expire.ts index 55c0463b..64ffb05e 100644 --- a/packages/cli/src/forget-ledger-expire.ts +++ b/packages/cli/src/forget-ledger-expire.ts @@ -4,7 +4,7 @@ import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { dirname, join, resolve } from "node:path"; import { promisify } from "node:util"; -import type { KvLedgerKey } from "@bounda-dev/cloudflare/forget-ledger"; +import type { KvLedgerKey, KvLedgerPut } from "@bounda-dev/cloudflare/forget-ledger"; const run = promisify(execFile); @@ -94,6 +94,15 @@ const isListedKey = (value: unknown): value is KvLedgerKey => typeof Reflect.get(value, "name") === "string" && ["number", "undefined"].includes(typeof Reflect.get(value, "expiration")); +const holdsEntry = (value: string): boolean => { + try { + const parsed: unknown = JSON.parse(value); + return typeof parsed === "object" && parsed !== null && !Array.isArray(parsed); + } catch { + return false; + } +}; + export const expireKvLedger: ExpireKvLedgerFunction = async ({ wrangler, ledger, @@ -123,7 +132,28 @@ export const expireKvLedger: ExpireKvLedgerFunction = async ({ if (!Array.isArray(listed) || !listed.every(isListedKey)) { throw new Error("wrangler kv key list printed something else than a list of keys"); } - const { expire, remove } = ledger.planKvForgetLedgerExpiry({ keys: listed, keepForMs, now }); + const plan = ledger.planKvForgetLedgerExpiry({ keys: listed, keepForMs, now }); + const { remove } = plan; + // A key whose value holds the entry is written again with that value, one read at a time; a + // value that is gone, which wrangler answers with a line of its own, leaves the key alone. + const expire: KvLedgerPut[] = []; + for (const put of plan.expire) { + if (put.value !== null) { + expire.push(put); + continue; + } + const value = await wrangler([ + "kv", + "key", + "get", + put.key, + "--binding", + binding, + "--text", + ...where, + ]); + if (holdsEntry(value)) expire.push({ ...put, value }); + } const directory = await mkdtemp(join(tmpdir(), "bounda-forget-ledger-")); try { if (expire.length > 0) { diff --git a/packages/cli/src/forgotten.test.ts b/packages/cli/src/forgotten.test.ts index 7b1ac54c..367565d1 100644 --- a/packages/cli/src/forgotten.test.ts +++ b/packages/cli/src/forgotten.test.ts @@ -64,10 +64,16 @@ describe("readForgotten", () => { reason: "retention", fingerprints: ["k-1"], }), - event(1_007, "bounda", "c-2", "PersonalDataForgotten", "1970-01-01T00:00:00.001Z"), + event(1_007, "bounda", "c-2", "PersonalDataForgotten", "1970-01-01T00:00:00.001Z", { + whole: false, + categories: ["marketing"], + retained: ["invoices"], + }), event(1_008, "bounda", "c-3", "PersonalDataForgotten", undefined, { reason: "restored", fingerprints: [1], + whole: false, + categories: "marketing", }), event(1_009, "bounda", "c-4", "PersonalDataForgotten", undefined, null), event(1_010, "bounda", "c-5", "PersonalDataForgotten", undefined, "text"), @@ -82,6 +88,7 @@ describe("readForgotten", () => { personId: "c-1", eventId: "e-1006", forgottenAt: "2026-11-02T00:00:00.000Z", + categories: null, reason: "retention", fingerprints: ["k-1"], }, @@ -89,18 +96,21 @@ describe("readForgotten", () => { personId: "c-2", eventId: "e-1007", forgottenAt: "1970-01-01T00:00:00.001Z", + categories: ["marketing", "invoices"], reason: "requested", }, { personId: "c-3", eventId: "e-1008", forgottenAt: "2026-11-01T00:00:00.000Z", + categories: [], reason: "requested", }, ...["c-4", "c-5"].map((personId, index) => ({ personId, eventId: `e-${1_009 + index}`, forgottenAt: "2026-11-01T00:00:00.000Z", + categories: null, reason: "requested", })), ]); @@ -128,12 +138,14 @@ describe("parseForgetList", () => { it("reads entries, bare ids and blank lines", () => { expect( parseForgetList( - '{"personId":"c-1","eventId":"e-1","forgottenAt":"x"}\r\n\n c-2 \n{"personId":"c-3"}\n', + '{"personId":"c-1","eventId":"e-1","forgottenAt":"x"}\r\n\n c-2 \n{"personId":"c-3"}\n{"personId":"c-4","categories":null}\n{"personId":"c-5","categories":["marketing"]}\n', ), ).toStrictEqual([ { personId: "c-1", eventId: "e-1", forgottenAt: "x" }, { personId: "c-2" }, { personId: "c-3" }, + { personId: "c-4", categories: null }, + { personId: "c-5", categories: ["marketing"] }, ]); }); @@ -163,9 +175,11 @@ describe("parseForgetList", () => { '{"personId":"c-1","reason":"toString"}', '{"personId":"c-1","fingerprints":"k-1"}', '{"personId":"c-1","fingerprints":["k-1",2]}', + '{"personId":"c-1","categories":"marketing"}', + '{"personId":"c-1","categories":[1]}', ]) { expect(() => parseForgetList(`c-0\n${line}`)).toThrow( - 'line 2 needs a "personId" string; "eventId" and "forgottenAt" only as strings, "reason" only as "requested" or "retention", and "fingerprints" only as a list of strings', + 'line 2 needs a "personId" string; "eventId" and "forgottenAt" only as strings, "reason" only as "requested" or "retention", "fingerprints" only as a list of strings, and "categories" only as null or a list of names', ); } expect(() => parseForgetList("{not json")).toThrow("line 1 is neither a person's id nor JSON"); diff --git a/packages/cli/src/forgotten.ts b/packages/cli/src/forgotten.ts index 5d8cf29a..ca893e7c 100644 --- a/packages/cli/src/forgotten.ts +++ b/packages/cli/src/forgotten.ts @@ -60,6 +60,15 @@ const REASONS: Readonly> = { const isReason = (value: unknown): value is PersonalDataForgottenReason => typeof value === "string" && Object.hasOwn(REASONS, value); +// What the forgetting asked for: `null` for the person as a whole, as one that does not say it +// forgot only some categories did, else the categories named, those retained included. +const askedOf = (payload: unknown): readonly string[] | null => { + if (fieldOf(payload, "whole") !== false) return null; + const forgotten = fieldOf(payload, "categories"); + const retained = fieldOf(payload, "retained"); + return [...(isTextList(forgotten) ? forgotten : []), ...(isTextList(retained) ? retained : [])]; +}; + export interface ReadForgottenArgs { readonly events: GlobalStream; readonly since: Since; @@ -87,6 +96,7 @@ export const readForgotten: ReadForgottenFunction = async ({ events, since }) => personId: event.aggregateId, eventId: event.id, forgottenAt: event.timestamp, + categories: askedOf(event.payload), reason: isReason(reason) ? reason : "requested", ...(isTextList(fingerprints) ? { fingerprints } : {}), }); @@ -110,15 +120,17 @@ const entryOf = (line: string, number: number): ForgetAgainEntry => { const forgottenAt = fieldOf(parsed, "forgottenAt"); const reason = fieldOf(parsed, "reason"); const fingerprints = fieldOf(parsed, "fingerprints"); + const categories = fieldOf(parsed, "categories"); if ( typeof personId !== "string" || (eventId !== undefined && typeof eventId !== "string") || (forgottenAt !== undefined && typeof forgottenAt !== "string") || (reason !== undefined && !isReason(reason)) || - (fingerprints !== undefined && !isTextList(fingerprints)) + (fingerprints !== undefined && !isTextList(fingerprints)) || + (categories !== undefined && categories !== null && !isTextList(categories)) ) { throw new Error( - `line ${number} needs a "personId" string; "eventId" and "forgottenAt" only as strings, "reason" only as "requested" or "retention", and "fingerprints" only as a list of strings`, + `line ${number} needs a "personId" string; "eventId" and "forgottenAt" only as strings, "reason" only as "requested" or "retention", "fingerprints" only as a list of strings, and "categories" only as null or a list of names`, ); } return { @@ -127,6 +139,7 @@ const entryOf = (line: string, number: number): ForgetAgainEntry => { ...(forgottenAt === undefined ? {} : { forgottenAt }), ...(reason === undefined ? {} : { reason }), ...(fingerprints === undefined ? {} : { fingerprints }), + ...(categories === undefined ? {} : { categories }), }; }; diff --git a/packages/cli/src/generate/discover.ts b/packages/cli/src/generate/discover.ts index 71477ac6..ac03da3f 100644 --- a/packages/cli/src/generate/discover.ts +++ b/packages/cli/src/generate/discover.ts @@ -1085,6 +1085,8 @@ const checkProjectionEvents = async ( const AUDIT_FILE = "audit.ts"; +const PERSONAL_DATA_FILE = "personal-data.ts"; + // No module is imported: the only text read is what the modules at an aggregate's or a read // model's root export, whether a policy or a projection exports `on`, and the interface a port // declares. @@ -1128,12 +1130,14 @@ export const discoverProject: DiscoverProjectFunction = async ({ root, appDir = problems.throwIfAny(); await checkProjectionEvents(context, aggregates, readModels); const audit = join(app, AUDIT_FILE); + const personalData = join(app, PERSONAL_DATA_FILE); return { root, appDir, aggregates: await resolvePolicyTriggers(context, aggregates), readModels, ...((await exists(audit)) ? { audit: moduleRef(context, audit) } : {}), + ...((await exists(personalData)) ? { personalData: moduleRef(context, personalData) } : {}), warnings: context.warnings, }; }; diff --git a/packages/cli/src/generate/emit/emit.test.ts b/packages/cli/src/generate/emit/emit.test.ts index 6a12c97f..1cd7d97b 100644 --- a/packages/cli/src/generate/emit/emit.test.ts +++ b/packages/cli/src/generate/emit/emit.test.ts @@ -447,6 +447,8 @@ export type ReactionCommands = core.ReactionCommandsFacadeOf = Check; +export type PersonalCategory = never; + export type ShipmentsRow = core.RowOf; export type ShipmentsFieldTypes = core.FieldTypesOf; export type ShipmentsPorts = core.EmptyPayload; diff --git a/packages/cli/src/generate/emit/plus-types.ts b/packages/cli/src/generate/emit/plus-types.ts index a535d774..d7b8b5ec 100644 --- a/packages/cli/src/generate/emit/plus-types.ts +++ b/packages/cli/src/generate/emit/plus-types.ts @@ -311,6 +311,17 @@ const readModelFiles = ( * core with the maps from `.bounda/types.ts`. The same template always renders the same way: * these files are canonical output, not subject to the project's formatter. */ +const personalDataFiles = (model: ProjectModel, typesPath: string): GeneratedFile[] => + model.personalData === undefined + ? [] + : [ + render(plusTypesPath(model.personalData.path), typesPath, { + imports: { generated: false, module: null }, + namespace: "PersonalData", + members: [["Categories", "core.PersonalDataCategories"]], + }), + ]; + export const emitPlusTypes: EmitPlusTypesFunction = ({ model, typesPath }) => [ ...model.aggregates.flatMap((aggregate) => [ ...eventFiles(aggregate, typesPath), @@ -319,4 +330,5 @@ export const emitPlusTypes: EmitPlusTypesFunction = ({ model, typesPath }) => [ ...processFiles(model, aggregate, typesPath), ]), ...model.readModels.flatMap((readModel) => readModelFiles(model, readModel, typesPath)), + ...personalDataFiles(model, typesPath), ]; diff --git a/packages/cli/src/generate/emit/register.ts b/packages/cli/src/generate/emit/register.ts index 0d0f4e89..cf33ae94 100644 --- a/packages/cli/src/generate/emit/register.ts +++ b/packages/cli/src/generate/emit/register.ts @@ -1,5 +1,9 @@ import type { GeneratedFile } from "./paths.ts"; -import { PORTS_CONFIG_TYPE_NAME, TEST_PORTS_TYPE_NAME } from "./types.ts"; +import { + PERSONAL_CATEGORY_TYPE_NAME, + PORTS_CONFIG_TYPE_NAME, + TEST_PORTS_TYPE_NAME, +} from "./types.ts"; export interface EmitRegisterArgs { readonly path: string; @@ -10,22 +14,24 @@ export interface EmitRegisterFunction { } /** - * Registers the registry type and the types of the `ports` configuration and of - * `createTestApp`'s `ports` with `@bounda-dev/core/register`, so `BoundaApp`, `boot()` and - * the integrations are typed for the project without a type argument and `defineConfig` and - * `createTestApp` check the implementation names. + * Registers the registry type, the types of the `ports` configuration and of `createTestApp`'s + * `ports`, and the names of the categories of personal data with `@bounda-dev/core/register`, so + * `BoundaApp`, `boot()` and the integrations are typed for the project without a type argument, + * `defineConfig` and `createTestApp` check the implementation names, and `personal({ category })` + * and `forget` check the categories. */ export const emitRegister: EmitRegisterFunction = ({ path }) => ({ path, content: [ 'import type { registry } from "./registry.ts";', - `import type { ${PORTS_CONFIG_TYPE_NAME}, ${TEST_PORTS_TYPE_NAME} } from "./types.ts";`, + `import type { ${PERSONAL_CATEGORY_TYPE_NAME}, ${PORTS_CONFIG_TYPE_NAME}, ${TEST_PORTS_TYPE_NAME} } from "./types.ts";`, "", 'declare module "@bounda-dev/core/register" {', " interface Register {", " readonly registry: typeof registry;", ` readonly ports: ${PORTS_CONFIG_TYPE_NAME};`, ` readonly testPorts: ${TEST_PORTS_TYPE_NAME};`, + ` readonly personalCategories: ${PERSONAL_CATEGORY_TYPE_NAME};`, " }", "}", "", diff --git a/packages/cli/src/generate/emit/registry.ts b/packages/cli/src/generate/emit/registry.ts index f04e129f..f892afc4 100644 --- a/packages/cli/src/generate/emit/registry.ts +++ b/packages/cli/src/generate/emit/registry.ts @@ -75,6 +75,7 @@ const collectImports = (model: ProjectModel): readonly ImportEntry[] => { for (const query of readModel.queries) add(query.key, readModel.name, query.path); } if (model.audit !== undefined) add("audit", "app", model.audit.path); + if (model.personalData !== undefined) add("personalData", "app", model.personalData.path); return entries; }; @@ -246,7 +247,8 @@ export interface EmitRegistryFunction { /** * `.bounda/registry.ts`: one namespace import per module, a type import per port, and the - * structured registry `createApp` consumes, with `app/audit.ts` when the project has one. + * structured registry `createApp` consumes, with `app/audit.ts` and `app/personal-data.ts` when + * the project has them. */ export const emitRegistry: EmitRegistryFunction = ({ model, path, fingerprints = {} }) => { const aliases = resolveAliases(model, path); @@ -273,6 +275,9 @@ export const emitRegistry: EmitRegistryFunction = ({ model, path, fingerprints = ...(model.audit === undefined ? [] : [` ${record([["audit", aliases.of(model.audit.path)]]).slice(2, -2)},`]), + ...(model.personalData === undefined + ? [] + : [` ${record([["personalData", aliases.of(model.personalData.path)]]).slice(2, -2)},`]), "} as const satisfies Registry;", "", ].join("\n"); diff --git a/packages/cli/src/generate/emit/types.ts b/packages/cli/src/generate/emit/types.ts index 10503c28..20e73b6a 100644 --- a/packages/cli/src/generate/emit/types.ts +++ b/packages/cli/src/generate/emit/types.ts @@ -91,6 +91,10 @@ const QUERIES_TYPE_NAME = "Queries"; export const TEST_PORTS_TYPE_NAME = "TestPorts"; +export const PERSONAL_CATEGORY_TYPE_NAME = "PersonalCategory"; + +const PERSONAL_DATA_CHECK_TYPE_NAME = "PersonalDataDeclaration"; + /** * The types `.bounda/types.ts` declares once for the whole app, next to those of each aggregate * and read model. @@ -102,6 +106,8 @@ export const APP_TYPE_NAMES: readonly string[] = [ QUERIES_TYPE_NAME, PORTS_CONFIG_TYPE_NAME, TEST_PORTS_TYPE_NAME, + PERSONAL_CATEGORY_TYPE_NAME, + PERSONAL_DATA_CHECK_TYPE_NAME, ]; const typeofImport = (from: string, to: string): string => @@ -264,6 +270,18 @@ const emitForgetting = (model: ProjectModel, readModel: ReadModelModel, path: st ].join("\n"); }; +/** + * The categories `app/personal-data.ts` declares, `never` without it, and a type that fails to + * compile while it declares `default` or a category that is not one. + */ +const emitPersonalData = (model: ProjectModel, path: string): string => + model.personalData === undefined + ? `export type ${PERSONAL_CATEGORY_TYPE_NAME} = never;` + : [ + `export type ${PERSONAL_CATEGORY_TYPE_NAME} = core.PersonalCategoriesOf<${typeofImport(path, model.personalData.path)}>;`, + `export type ${PERSONAL_DATA_CHECK_TYPE_NAME} = ${EXPECT_TYPE_NAME}>;`, + ].join("\n"); + const emitMap = ( name: string, wrapper: string, @@ -312,9 +330,10 @@ export const emitTypes: EmitTypesFunction = ({ model, path, inferredStates = {} ); sections.push(emitMap(COMMANDS_TYPE_NAME, "CommandsFacadeOf", commandModules)); sections.push(emitMap(REACTION_COMMANDS_TYPE_NAME, "ReactionCommandsFacadeOf", commandModules)); - if (model.readModels.length > 0) { + if (model.readModels.length > 0 || model.personalData !== undefined) { sections.push(`type ${EXPECT_TYPE_NAME} = Check;`); } + sections.push(emitPersonalData(model, path)); for (const readModel of model.readModels) { sections.push( [ diff --git a/packages/cli/src/generate/model.ts b/packages/cli/src/generate/model.ts index d3be3360..3804f2fa 100644 --- a/packages/cli/src/generate/model.ts +++ b/packages/cli/src/generate/model.ts @@ -109,4 +109,6 @@ export interface ProjectModel { readonly readModels: readonly ReadModelModel[]; // `app/audit.ts`, when the project has one. readonly audit?: ModuleRef; + // `app/personal-data.ts`, when the project has one. + readonly personalData?: ModuleRef; } diff --git a/packages/cli/src/run.test.ts b/packages/cli/src/run.test.ts index be034f0f..77b0db27 100644 --- a/packages/cli/src/run.test.ts +++ b/packages/cli/src/run.test.ts @@ -2,7 +2,7 @@ import { writeFileSync } from "node:fs"; import { chmod, cp, mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { basename, join, resolve } from "node:path"; -import { createApp, silentLogger } from "@bounda-dev/core"; +import { createApp, type PersonalDataExport, silentLogger } from "@bounda-dev/core"; import { loadProject } from "@bounda-dev/core/node"; import { afterAll, describe, expect, it, vi } from "vitest"; import type { registry as failingRegistry } from "./fixtures/rebuild-project/.bounda/failing-registry.ts"; @@ -714,6 +714,7 @@ describe("bounda forgotten and bounda forget", () => { personId: forgotten, eventId: event?.id, forgottenAt: event?.timestamp, + categories: null, reason: "requested", fingerprints: Reflect.get(Object(event?.payload), "fingerprints"), })}\n`, @@ -744,6 +745,47 @@ describe("bounda forgotten and bounda forget", () => { ]); }); + it("forgets a person, or some categories of theirs, and exports what the store holds of them", async () => { + const memberId = `m-${Date.now()}-direct`; + const app = await open(); + await app.commands.join({ memberId, name: "Ivo" }); + await app.stop(); + + const exported = await cli(["export", memberId, "--actor", "operator-1"], fixture); + expect(exported.code).toBe(EXIT_OK); + const data = JSON.parse(exported.stdout) as PersonalDataExport; + expect(data.personId).toBe(memberId); + expect(data.categories.default).toMatchObject({ basis: null, restrictedUntil: null }); + expect(data.events.length).toBeGreaterThan(0); + + const unknown = await cli(["forget", memberId, "--category", "marketing"], fixture); + expect(unknown.code).not.toBe(EXIT_OK); + const partial = await cli(["forget", memberId, "--category", "default"], fixture); + expect(partial).toMatchObject({ code: EXIT_OK, stdout: `forgot default of ${memberId}\n` }); + const whole = await cli(["forget", memberId, "--actor", "operator-1"], fixture); + expect(whole).toMatchObject({ code: EXIT_OK, stdout: `forgot ${memberId}\n` }); + const after = await open(); + const forgettings = (await after.events.read({ limit: 1_000 })) + .filter((event) => event.type === "PersonalDataForgotten" && event.aggregateId === memberId) + .map(({ payload, metadata }) => [(payload as { whole: boolean }).whole, metadata.actor]); + await after.stop(); + expect(forgettings).toEqual([ + [false, undefined], + [true, "operator-1"], + ]); + }); + + it("refuses a forget that names no one, or both a person and a list", async () => { + const neither = await cli(["forget"], fixture); + expect(neither.code).toBe(EXIT_CONVENTION); + expect(neither.stderr).toContain("name one person, or a list with --from, not both"); + const both = await cli(["forget", "m-1", "--from", "list.txt"], fixture); + expect(both.code).toBe(EXIT_CONVENTION); + const listed = await cli(["forget", "--from", "list.txt", "--category", "default"], fixture); + expect(listed.code).toBe(EXIT_CONVENTION); + expect(listed.stderr).toContain("--category forgets one person"); + }); + it("names who forgets again with --actor, in the event's metadata", async () => { const memberId = `m-${Date.now()}-acted`; const app = await open(); diff --git a/packages/cli/src/run.ts b/packages/cli/src/run.ts index 74dbf69e..9156d11b 100644 --- a/packages/cli/src/run.ts +++ b/packages/cli/src/run.ts @@ -311,6 +311,22 @@ const runDiscardDeadLetter = ( return EXIT_OK; }); +// Printed whole: it is what the person asked for, and goes where the operator sends stdout. +const runExport = ( + personId: string, + options: ActingOptions, + cwd: string, + stdout: Output, + stderr: Output, +): Promise => + withApp(options, cwd, stderr, async (app) => { + line( + stdout, + JSON.stringify(await actingAs(app, options).exportPersonalData(personId), null, 2), + ); + return EXIT_OK; + }); + interface ForgottenOptions extends ProjectOptions { readonly since: Since; } @@ -355,24 +371,60 @@ const runForgotten = async ( }; interface ForgetOptions extends ActingOptions { - readonly from: string; + readonly from?: string; + readonly category?: readonly string[]; } +const collect = (value: string, previous: readonly string[] = []): readonly string[] => [ + ...previous, + value, +]; + +const runForgetPerson = ( + personId: string, + options: ForgetOptions, + cwd: string, + stdout: Output, + stderr: Output, +): Promise => + withApp(options, cwd, stderr, async (app) => { + const categories = options.category; + await actingAs(app, options).forget( + personId, + categories === undefined ? undefined : { categories }, + ); + line( + stdout, + categories === undefined + ? `forgot ${personId}` + : `forgot ${categories.join(", ")} of ${personId}`, + ); + return EXIT_OK; + }); + const runForget = async ( + personId: string | undefined, options: ForgetOptions, cwd: string, stdout: Output, stderr: Output, ): Promise => { + if ((personId === undefined) === (options.from === undefined)) { + line(stderr, "error: name one person, or a list with --from, not both"); + return EXIT_CONVENTION; + } + if (personId !== undefined) return runForgetPerson(personId, options, cwd, stdout, stderr); + if (options.category !== undefined) { + line(stderr, "error: --category forgets one person; a list says what each line asked for"); + return EXIT_CONVENTION; + } + const from = options.from as string; let entries: ReturnType; try { - entries = parseForgetList(await readFile(resolve(cwd, options.from), "utf8")); + entries = parseForgetList(await readFile(resolve(cwd, from), "utf8")); } catch (error) { // Its own file, read and parsed here: what fails names the path or the line, never a value. - line( - stderr, - `error: ${options.from}: ${error instanceof Error ? error.message : String(error)}`, - ); + line(stderr, `error: ${from}: ${error instanceof Error ? error.message : String(error)}`); return EXIT_FAILURE; } return withApp(options, cwd, stderr, async (app) => { @@ -698,11 +750,28 @@ export const runCli: RunCliFunction = async ({ program .command("forget") .description( - "forget again, after restoring a backup, each person listed who still holds a key and whose forgetting the store does not hold", + "forget a person, or some categories of theirs; with --from, forget again, after restoring a backup, each person listed who still holds a key and whose forgetting the store does not hold", + ) + .argument("[person]", "the id of the person to forget") + .option( + "--category ", + "forget only this category of the person, as when a consent is withdrawn; repeat it for several", + collect, + ) + .option("--from ", "what bounda forgotten printed, or one person's id a line"), + ).action(async (personId: string | undefined, options: ForgetOptions) => { + exitCode = await runForget(personId, options, cwd, stdout, stderr); + }); + + actingOptions( + program + .command("export") + .description( + "print, as JSON, everything the store holds that names a person, with each field's category, basis and retention; recorded in the access log", ) - .requiredOption("--from ", "what bounda forgotten printed, or one person's id a line"), - ).action(async (options: ForgetOptions) => { - exitCode = await runForget(options, cwd, stdout, stderr); + .argument("", "the id of the person to export"), + ).action(async (personId: string, options: ActingOptions) => { + exitCode = await runExport(personId, options, cwd, stdout, stderr); }); program diff --git a/packages/cli/src/status-output.test.ts b/packages/cli/src/status-output.test.ts index 2b8de18e..c39f4aa5 100644 --- a/packages/cli/src/status-output.test.ts +++ b/packages/cli/src/status-output.test.ts @@ -55,7 +55,13 @@ const status: AppStatus = { held: { until: "2026-05-01T00:05:00.000Z", attempts: 2 }, }, keyRotation: { kid: "k-2", previous: ["k-1"], pending: 1, completeAt: null }, - retention: { retentionMs: 30 * 86_400_000, due: 1, nextDueAt: "2026-05-03T00:00:00.000Z" }, + retention: { + categories: [ + { category: "default", retention: "30d", due: 1, nextDueAt: "2026-05-03T00:00:00.000Z" }, + { category: "invoices", retention: "6y", due: 2, nextDueAt: null }, + ], + keptBeyond: 1, + }, deadLetterRetention: { retentionMs: 90_000, nextDueAt: null }, }; @@ -75,7 +81,10 @@ describe("formatStatus", () => { "personal data backfill: sealed through position 9, not caught up yet", "forget ledger: not reconciled since this instance started, applied through 2026-04-30T00:00:00.000Z, last run never, held until 2026-05-01T00:05:00.000Z after 2 failures", "key rotation: under k-2 (previous k-1), 1 key left to rewrap", - "retention: 30d, 1 person past it, next due 2026-05-03T00:00:00.000Z", + "retention", + " default 30d, 1 person past it, next due 2026-05-03T00:00:00.000Z", + " invoices 6y, 2 keys past it, next due never", + " 1 key kept past a forgetting that no legal obligation keeps", "dead letter retention: 90s, next purge when a letter is resolved", ]); }); @@ -110,9 +119,12 @@ describe("formatStatus", () => { "personal data backfill: no personal data marked", "forget ledger: reconciled, applied through nothing yet, last run 2026-05-01T00:00:00.000Z", "key rotation: under k-1, every key wrapped (checked 2026-05-01T00:00:00.000Z)", - "retention: not configured", + "retention: no personal data marked", "dead letter retention: not configured", ]); + expect( + formatStatus({ status: { ...bare, retention: { categories: [], keptBeyond: 0 } } }), + ).toContain("retention: not configured\n"); expect( formatStatus({ status: { ...bare, keyRotation: { ...bare.keyRotation, completeAt: null } as never }, diff --git a/packages/cli/src/status-output.ts b/packages/cli/src/status-output.ts index 16f6ef53..97d9093c 100644 --- a/packages/cli/src/status-output.ts +++ b/packages/cli/src/status-output.ts @@ -1,4 +1,5 @@ import type { AppStatus, SchemaStatus, SubscriberFailure } from "@bounda-dev/core"; +import { DEFAULT_PERSONAL_CATEGORY } from "@bounda-dev/core/adapter"; export interface FormatStatusArgs { readonly status: AppStatus; @@ -83,10 +84,22 @@ const rotationOf = ({ keyRotation: rotation }: AppStatus): string => { return `under ${rotation.kid}${rotation.previous.length === 0 ? "" : ` (previous ${rotation.previous.join(", ")})`}, ${left}`; }; -const retentionOf = ({ retention }: AppStatus): string => - retention === null - ? "not configured" - : `${days(retention.retentionMs)}, ${plural(retention.due, "person")} past it, next due ${retention.nextDueAt ?? "never"}`; +// One line per retention set: the app's forgets persons as a whole, a category's its keys. +const retentionOf = ({ retention }: AppStatus): string[] => { + if (retention === null) return ["retention: no personal data marked"]; + const { categories, keptBeyond } = retention; + if (categories.length === 0 && keptBeyond === 0) return ["retention: not configured"]; + return [ + "retention", + ...categories.map( + ({ category, retention: kept, due, nextDueAt }) => + ` ${category} ${kept}, ${plural(due, category === DEFAULT_PERSONAL_CATEGORY ? "person" : "key")} past it, next due ${nextDueAt ?? "never"}`, + ), + ...(keptBeyond === 0 + ? [] + : [` ${plural(keptBeyond, "key")} kept past a forgetting that no legal obligation keeps`]), + ]; +}; const purgeOf = ({ deadLetterRetention: purge }: AppStatus): string => purge === null @@ -107,7 +120,7 @@ export const formatStatus: FormatStatusFunction = (args) => { `personal data backfill: ${backfillOf(status)}`, `forget ledger: ${ledgerOf(status)}`, `key rotation: ${rotationOf(status)}`, - `retention: ${retentionOf(status)}`, + ...retentionOf(status), `dead letter retention: ${purgeOf(status)}`, ].join("\n"); }; diff --git a/packages/cloudflare/src/bounda-object.ts b/packages/cloudflare/src/bounda-object.ts index 078cb9b5..92249acc 100644 --- a/packages/cloudflare/src/bounda-object.ts +++ b/packages/cloudflare/src/bounda-object.ts @@ -12,6 +12,7 @@ import { type DispatchOptions, type DispatchResult, errorFields, + type ForgetOptions, type IdGenerator, type ListDeadLettersArgs, type Logger, @@ -409,8 +410,9 @@ export const createBoundaObject: CreateBoundaObjectFunction = > { - return this.#onPerson((client) => client.forget(personId), consistency, actor); + return this.#onPerson((client) => client.forget(personId, options), consistency, actor); } restrict( diff --git a/packages/cloudflare/src/client.ts b/packages/cloudflare/src/client.ts index 4af30892..b49886df 100644 --- a/packages/cloudflare/src/client.ts +++ b/packages/cloudflare/src/client.ts @@ -10,6 +10,7 @@ import { type DeadLetter, type DispatchOptions, type DispatchResult, + type ForgetOptions, type ListDeadLettersArgs, type PersonalDataExport, type QueriesFacade, @@ -45,7 +46,12 @@ export interface BoundaStub { retryDeadLetter(id: string, actor?: string): PromiseLike; discardDeadLetter(id: string, actor?: string): PromiseLike; rebuildReadModel(name: string): PromiseLike; - forget(personId: string, consistency?: Consistency, actor?: string): PromiseLike; + forget( + personId: string, + consistency?: Consistency, + actor?: string, + options?: ForgetOptions, + ): PromiseLike; exportPersonalData(personId: string, actor?: string): PromiseLike; restrict(personId: string, consistency?: Consistency, actor?: string): PromiseLike; unrestrict(personId: string, consistency?: Consistency, actor?: string): PromiseLike; @@ -127,7 +133,8 @@ export const connect: ConnectFunction = ( discard: (id) => unwrap(stub.discardDeadLetter(id, actor)), }, rebuildReadModel: (name) => unwrap(stub.rebuildReadModel(name)), - forget: (personId) => unwrap(stub.forget(personId, consistency, actor)), + forget: (personId, options) => + unwrap(stub.forget(personId, consistency, actor, options)), exportPersonalData: (personId) => unwrap(stub.exportPersonalData(personId, actor)), restrict: (personId) => unwrap(stub.restrict(personId, consistency, actor)), diff --git a/packages/cloudflare/src/forget-ledger.ts b/packages/cloudflare/src/forget-ledger.ts index 27a768fd..3a5ea05c 100644 --- a/packages/cloudflare/src/forget-ledger.ts +++ b/packages/cloudflare/src/forget-ledger.ts @@ -18,17 +18,84 @@ const LIST_LIMIT = 1_000; const MARKS_PREFIX = "bounda/forget-ledger-marks/"; // The soonest KV lets a key expire, in seconds from now. const KV_MIN_TTL_S = 60; +// The most KV takes as a key's metadata, in bytes of its JSON. +const KV_METADATA_BYTES = 1_024; +// `categories` is left out for a forgetting of the person as a whole, as every entry written +// before categories was. interface EntryMetadata { readonly personId: string; readonly forgottenAt: string; + readonly categories?: readonly string[]; // Absent from a key written before entries kept them. readonly reason?: string; readonly fingerprints?: readonly string[]; + // The categories and the fingerprints are the key's value instead, as JSON: they did not fit. + readonly inValue?: true; +} + +// What the value of an entry `inValue` holds. +interface SpilledEntry { + readonly categories: readonly string[] | null; + readonly fingerprints?: readonly string[]; +} + +interface StoredEntry { + readonly metadata: EntryMetadata; + readonly value: string; } const timeKey = (forgottenAt: string): string => String(LATEST_MS - Date.parse(forgottenAt)); +const sizeOf = (metadata: EntryMetadata): number => + new TextEncoder().encode(JSON.stringify(metadata)).length; + +// A list reads the metadata alone, so an entry is kept there, but for the categories and keys of +// a person with too many to fit, which a read then fetches from the key's value. +const storedOf = ({ + personId, + forgottenAt, + categories, + reason, + fingerprints, +}: ForgetLedgerEntry): StoredEntry => { + const full: EntryMetadata = { + personId, + forgottenAt, + ...(categories === null ? {} : { categories }), + reason, + ...(fingerprints === undefined ? {} : { fingerprints }), + }; + if (sizeOf(full) <= KV_METADATA_BYTES) return { metadata: full, value: "" }; + const spilled: SpilledEntry = { + categories, + ...(fingerprints === undefined ? {} : { fingerprints }), + }; + return { + metadata: { personId, forgottenAt, reason, inValue: true }, + value: JSON.stringify(spilled), + }; +}; + +const isTextList = (value: unknown): value is readonly string[] => + Array.isArray(value) && value.every((item) => typeof item === "string"); + +// What a value holds of its entry, `null` when it is gone or holds something else: the entry is +// then left out, rather than read as a forgetting of the person as a whole. +const spilledIn = (value: string | null): SpilledEntry | null => { + let parsed: unknown; + try { + parsed = JSON.parse(value ?? "null"); + } catch { + return null; + } + if (typeof parsed !== "object" || parsed === null) return null; + const categories: unknown = Reflect.get(parsed, "categories"); + const fingerprints: unknown = Reflect.get(parsed, "fingerprints"); + if (categories !== null && !isTextList(categories)) return null; + return isTextList(fingerprints) ? { categories, fingerprints } : { categories }; +}; + /** * Where every object's entries live, each under its own name. */ @@ -41,18 +108,28 @@ const expirationOf = (forgottenAt: string, keepForMs: number, now: number): numb return Math.max(Math.ceil(expiresAt / 1000), Math.floor(now / 1000) + KV_MIN_TTL_S); }; -const entryOf = (metadata: EntryMetadata, eventId: string): ForgetLedgerEntry => { +const entryOf = ( + metadata: EntryMetadata, + eventId: string, + { categories, fingerprints }: Partial = metadata, +): ForgetLedgerEntry => { const entry: ForgetLedgerEntry = { personId: metadata.personId, + categories: isTextList(categories) ? [...categories] : null, eventId, forgottenAt: metadata.forgottenAt, reason: metadata.reason === "retention" ? "retention" : "requested", }; - return Array.isArray(metadata.fingerprints) - ? { ...entry, fingerprints: [...metadata.fingerprints] } - : entry; + return isTextList(fingerprints) ? { ...entry, fingerprints: [...fingerprints] } : entry; }; +interface ListedEntry { + readonly name: string; + readonly metadata: EntryMetadata; + readonly forgottenAt: string; + readonly eventId: string; +} + export interface CreateKvForgetLedgerArgs { readonly namespace: KVNamespace; /** @@ -89,23 +166,19 @@ export const createKvForgetLedger: CreateKvForgetLedgerFunction = ({ append: async (entries) => { const now = Date.now(); await Promise.all( - entries.flatMap(({ personId, eventId, forgottenAt, reason, fingerprints }) => { - const metadata: EntryMetadata = { - personId, - forgottenAt, - reason, - ...(fingerprints === undefined ? {} : { fingerprints }), - }; + entries.flatMap((entry) => { + const { eventId, forgottenAt } = entry; + const { metadata, value } = storedOf(entry); const key = `${prefix}${timeKey(forgottenAt)}:${eventId}`; - if (keepForMs === undefined) return [namespace.put(key, "", { metadata })]; + if (keepForMs === undefined) return [namespace.put(key, value, { metadata })]; const expiration = expirationOf(forgottenAt, keepForMs, now); - return expiration === null ? [] : [namespace.put(key, "", { metadata, expiration })]; + return expiration === null ? [] : [namespace.put(key, value, { metadata, expiration })]; }), ); }, read: async ({ after, limit }) => { const oldest = after === undefined ? null : timeKey(after.forgottenAt); - const found: ForgetLedgerEntry[] = []; + const found: ListedEntry[] = []; let cursor: string | undefined; for (;;) { const page = await namespace.list({ @@ -122,15 +195,29 @@ export const createKvForgetLedger: CreateKvForgetLedgerFunction = ({ break; } if (metadata === null || metadata === undefined) continue; - found.push(entryOf(metadata, rest.slice(separator + 1))); + found.push({ + name, + metadata, + forgottenAt: metadata.forgottenAt, + eventId: rest.slice(separator + 1), + }); } if (past || page.list_complete) break; cursor = page.cursor; } - return found + const read = found .filter((entry) => after === undefined || byForgetLedgerOrder(entry, after) > 0) .sort(byForgetLedgerOrder) .slice(0, limit); + // Only the values of the entries handed out, at once. + const entries = await Promise.all( + read.map(async ({ name, metadata, eventId }) => { + if (metadata.inValue !== true) return entryOf(metadata, eventId); + const spilled = spilledIn(await namespace.get(name)); + return spilled === null ? null : entryOf(metadata, eventId, spilled); + }), + ); + return entries.filter((entry) => entry !== null); }, readMark: async (name) => Number((await namespace.get(markKey(name))) ?? 0), // KV has no compare: of two instances raising at once the lower may win, which the next raise, @@ -207,7 +294,11 @@ export interface KvLedgerKey { */ export interface KvLedgerPut { readonly key: string; - readonly value: ""; + /** + * `null` for a key whose value holds the entry's categories and keys: it is written again with + * the value it has, which the caller reads first, and left alone when that is gone. + */ + readonly value: string | null; readonly expiration: number; readonly metadata: unknown; } @@ -251,7 +342,8 @@ export const planKvForgetLedgerExpiry: PlanKvForgetLedgerExpiryFunction = ({ const expiration = expirationOf(forgottenAt, keepForMs, now); if (expiration === null) remove.push(name); else if (current === undefined || current > expiration) { - expire.push({ key: name, value: "", expiration, metadata }); + const inValue = Reflect.get(metadata as object, "inValue") === true; + expire.push({ key: name, value: inValue ? null : "", expiration, metadata }); } } return { expire, remove }; @@ -294,9 +386,10 @@ export const expireKvForgetLedger: ExpireKvForgetLedgerFunction = async ({ } const expiry = planKvForgetLedgerExpiry({ keys, keepForMs, now }); await Promise.all([ - ...expiry.expire.map(({ key, value, expiration, metadata }) => - namespace.put(key, value, { expiration, metadata }), - ), + ...expiry.expire.map(async ({ key, value, expiration, metadata }) => { + const kept = value ?? (await namespace.get(key)); + if (kept !== null) await namespace.put(key, kept, { expiration, metadata }); + }), ...expiry.remove.map((key) => namespace.delete(key)), ]); return expiry; diff --git a/packages/cloudflare/test/bounda-object.test.ts b/packages/cloudflare/test/bounda-object.test.ts index 32c7925f..66a86cf5 100644 --- a/packages/cloudflare/test/bounda-object.test.ts +++ b/packages/cloudflare/test/bounda-object.test.ts @@ -529,7 +529,7 @@ describe("a Bounda Durable Object", () => { outage.projection = false; const status = await store.status(); expect(status.schema).toEqual([ - { database: "storage", scope: "storage", version: 2, latest: 2 }, + { database: "storage", scope: "storage", version: 3, latest: 3 }, { database: "storage", scope: "checkpoints", version: 1, latest: 1 }, ]); expect(status.subscriberFailures).toContainEqual( diff --git a/packages/cloudflare/test/client.test.ts b/packages/cloudflare/test/client.test.ts index 221491d6..c91f8255 100644 --- a/packages/cloudflare/test/client.test.ts +++ b/packages/cloudflare/test/client.test.ts @@ -31,6 +31,7 @@ describe("connect", () => { const eventual = recordingStub(); const soon = connect(eventual.stub, { consistency: "eventual" }); await soon.forget("c-1"); + await soon.forget("c-1", { categories: ["default"] }); await soon.restrict("c-1"); await soon.unrestrict("c-1"); const byDefault = recordingStub(); @@ -40,12 +41,13 @@ describe("connect", () => { await store.unrestrict("c-2"); expect([eventual.forgets, byDefault.forgets]).toEqual([ [ - ["c-1", "eventual", undefined], + ["c-1", "eventual", undefined, undefined], + ["c-1", "eventual", undefined, { categories: ["default"] }], ["restrict", "c-1", "eventual", undefined], ["unrestrict", "c-1", "eventual", undefined], ], [ - ["c-2", "read-your-writes", undefined], + ["c-2", "read-your-writes", undefined, undefined], ["restrict", "c-2", "read-your-writes", undefined], ["unrestrict", "c-2", "read-your-writes", undefined], ], @@ -81,7 +83,7 @@ describe("connect", () => { ["getOrder", { orderId: "o-1" }, undefined], ], forgets: [ - ["c-1", "eventual", "user-1"], + ["c-1", "eventual", "user-1", undefined], ["restrict", "c-1", "eventual", "user-1"], ["unrestrict", "c-1", "eventual", "user-1"], ], diff --git a/packages/cloudflare/test/forget-ledger.test.ts b/packages/cloudflare/test/forget-ledger.test.ts index 0c806c08..206e3ed0 100644 --- a/packages/cloudflare/test/forget-ledger.test.ts +++ b/packages/cloudflare/test/forget-ledger.test.ts @@ -49,6 +49,7 @@ describe("the forget ledger in Workers KV", () => { personId: "c-1", eventId: "e-1", forgottenAt: "2026-11-07T12:00:00.000Z", + categories: ["marketing"], reason: "requested" as const, fingerprints: ["k-1"], }; @@ -118,6 +119,7 @@ describe("the KV forget ledger's listing", () => { personId: `c-${index}`, eventId: `e-${String(index).padStart(4, "0")}`, forgottenAt: minute(index), + categories: null, reason: "requested" as const, })); await ledger.append(entries); @@ -132,6 +134,7 @@ describe("the KV forget ledger's listing", () => { personId: `c-${index}`, eventId: `e-${String(index).padStart(4, "0")}`, forgottenAt: minute(index), + categories: null, reason: "requested" as const, })); await ledger.append(entries); @@ -151,14 +154,27 @@ describe("the KV forget ledger's listing", () => { const now = Date.now(); const at = (milliseconds: number): string => new Date(now + milliseconds).toISOString(); await ledger.append([ - { personId: "c-1", eventId: "e-1", forgottenAt: at(-keepForMs - 1_000), reason: "requested" }, + { + personId: "c-1", + eventId: "e-1", + forgottenAt: at(-keepForMs - 1_000), + categories: null, + reason: "requested", + }, { personId: "c-2", eventId: "e-2", forgottenAt: at(-keepForMs + 30_000), + categories: null, + reason: "requested", + }, + { + personId: "c-3", + eventId: "e-3", + forgottenAt: at(0), + categories: null, reason: "requested", }, - { personId: "c-3", eventId: "e-3", forgottenAt: at(0), reason: "requested" }, ]); expect( [...keys.values()].map((metadata) => (metadata as { personId: string }).personId), @@ -170,7 +186,13 @@ describe("the KV forget ledger's listing", () => { expect(later).toBe(Math.ceil((Date.parse(at(0)) + keepForMs) / 1000)); const forever = countingNamespace(); await createKvForgetLedger({ namespace: forever.namespace, prefix: "p/" }).append([ - { personId: "c-1", eventId: "e-1", forgottenAt: at(-keepForMs * 1_000), reason: "requested" }, + { + personId: "c-1", + eventId: "e-1", + forgottenAt: at(-keepForMs * 1_000), + categories: null, + reason: "requested", + }, ]); expect([...forever.expirations.values()]).toEqual([undefined]); }); @@ -182,6 +204,83 @@ describe("the KV forget ledger's listing", () => { expect(await ledger.read({ limit: 10 })).toEqual([]); }); + it("keeps an entry's categories and keys in its value when they do not fit KV's metadata", async () => { + const namespace = env.FORGET_LEDGER; + const ledger = createKvForgetLedger({ namespace, prefix: "overflow/" }); + const entry = (eventId: string, pad: string) => ({ + personId: "c-1", + categories: ["marketing", "invoices"], + eventId, + forgottenAt: minute(0), + reason: "requested" as const, + fingerprints: ["k-1", pad], + }); + const bytes = (value: unknown) => new TextEncoder().encode(JSON.stringify(value)).length; + const { eventId: _, ...unpadded } = entry("e-0", ""); + const pad = "x".repeat(1_024 - bytes(unpadded)); + const many = Array.from({ length: 100 }, (_, index) => `category-${index}`); + const { fingerprints: __, ...unkeyed } = entry("e-3", "k-2"); + await ledger.append([ + entry("e-1", pad), + entry("e-2", `${pad}x`), + { ...unkeyed, categories: many }, + { ...entry("e-4", "k-2"), categories: null }, + ]); + const { keys } = await namespace.list({ prefix: "overflow/" }); + expect( + keys.map(({ metadata }) => [bytes(metadata), Reflect.get(Object(metadata), "inValue")]), + ).toEqual([ + [1_024, undefined], + [ + bytes({ personId: "c-1", forgottenAt: minute(0), reason: "requested", inValue: true }), + true, + ], + [ + bytes({ personId: "c-1", forgottenAt: minute(0), reason: "requested", inValue: true }), + true, + ], + [expect.any(Number), undefined], + ]); + expect(JSON.parse((await namespace.get(keys[1]?.name ?? "")) ?? "")).toEqual({ + categories: ["marketing", "invoices"], + fingerprints: ["k-1", `${pad}x`], + }); + expect(await ledger.read({ limit: 10 })).toEqual([ + entry("e-1", pad), + entry("e-2", `${pad}x`), + { ...unkeyed, categories: many }, + { ...entry("e-4", "k-2"), categories: null }, + ]); + expect(await ledger.read({ limit: 2 })).toEqual([entry("e-1", pad), entry("e-2", `${pad}x`)]); + }); + + it("leaves out an entry whose value no longer holds it, rather than read it as a whole", async () => { + const { namespace, keys } = countingNamespace(); + const values = new Map([ + ["p/1:e-1", "not json"], + ["p/2:e-2", JSON.stringify({ categories: [1] })], + ["p/3:e-3", JSON.stringify("text")], + ["p/5:e-5", JSON.stringify({ categories: null, fingerprints: [2] })], + ]); + const reading = { + ...namespace, + get: async (name: string) => values.get(name) ?? null, + } as unknown as KVNamespace; + const ledger = createKvForgetLedger({ namespace: reading, prefix: "p/" }); + for (const at of ["1:e-1", "2:e-2", "3:e-3", "4:e-4", "5:e-5"]) { + keys.set(`p/${at}`, { personId: "c-1", forgottenAt: minute(0), inValue: true }); + } + expect(await ledger.read({ limit: 10 })).toEqual([ + { + personId: "c-1", + categories: null, + eventId: "e-5", + forgottenAt: minute(0), + reason: "requested", + }, + ]); + }); + it("reads a key written before entries kept their reason and keys as a request naming none", async () => { const { namespace, keys } = countingNamespace(); const ledger = createKvForgetLedger({ namespace, prefix: "p/" }); @@ -189,8 +288,20 @@ describe("the KV forget ledger's listing", () => { keys.set("p/2:e-2", { personId: "c-2", forgottenAt: minute(0), reason: "retention" }); const read = await ledger.read({ limit: 10 }); expect(read).toEqual([ - { personId: "c-1", eventId: "e-1", forgottenAt: minute(0), reason: "requested" }, - { personId: "c-2", eventId: "e-2", forgottenAt: minute(0), reason: "retention" }, + { + personId: "c-1", + categories: null, + eventId: "e-1", + forgottenAt: minute(0), + reason: "requested", + }, + { + personId: "c-2", + categories: null, + eventId: "e-2", + forgottenAt: minute(0), + reason: "retention", + }, ]); expect(read[0]).not.toHaveProperty("fingerprints"); }); @@ -219,10 +330,17 @@ describe("the KV forget ledger's listing", () => { personId: "c-1", eventId: "e-1", forgottenAt: minute(0), + categories: null, reason: "retention", fingerprints: ["k-1"], }, - { personId: "c-2", eventId: "e-2", forgottenAt: minute(0), reason: "requested" }, + { + personId: "c-2", + eventId: "e-2", + forgottenAt: minute(0), + categories: null, + reason: "requested", + }, ]); expect([...keys.keys()]).toEqual([ `${KV_FORGET_LEDGER_ROOT}acme%2Feu/${8_640_000_000_000_000 - Date.parse(minute(0))}:e-1`, @@ -331,6 +449,7 @@ describe("expiring a KV forget ledger", () => { { name: "sooner", metadata: at(2), expiration: NOW / 1000 + 86_400 }, { name: "due", metadata: at(2), expiration: (NOW + 33 * DAY) / 1000 }, { name: "soonest", metadata: soonest }, + { name: "valued", metadata: { ...at(1), inValue: true } }, { name: "stray" }, { name: "empty", metadata: null }, { name: "unreadable", metadata: { forgottenAt: "not a time" } }, @@ -350,6 +469,12 @@ describe("expiring a KV forget ledger", () => { expiration: NOW / 1000 + 60, metadata: soonest, }, + { + key: "valued", + value: null, + expiration: (NOW + 34 * DAY) / 1000, + metadata: { ...at(1), inValue: true }, + }, ], remove: ["past", "just-past"], }); @@ -366,7 +491,11 @@ describe("expiring a KV forget ledger", () => { await namespace.put(`${KV_FORGET_LEDGER_ROOT}expire-a/1:e-1`, "", { metadata: written(1) }); await namespace.put(`${KV_FORGET_LEDGER_ROOT}expire-b/2:e-2`, "", { metadata: written(40) }); await namespace.put("expire-elsewhere", "kept", { metadata: written(1) }); + const valued = `${KV_FORGET_LEDGER_ROOT}expire-c/3:e-3`; + const held = JSON.stringify({ categories: null, fingerprints: ["k-1"] }); + await namespace.put(valued, held, { metadata: { ...written(1), inValue: true } }); const expiry = await expireKvForgetLedger({ namespace, keepForMs: 35 * DAY, now }); + expect(await namespace.get(valued)).toBe(held); expect(expiry.expire.map(({ key }) => key)).toContain(`${KV_FORGET_LEDGER_ROOT}expire-a/1:e-1`); expect(expiry.remove).toContain(`${KV_FORGET_LEDGER_ROOT}expire-b/2:e-2`); const listed = await namespace.list({ prefix: KV_FORGET_LEDGER_ROOT }); @@ -383,6 +512,20 @@ describe("expiring a KV forget ledger", () => { }); }); + it("leaves alone a key whose value held its keys once it is gone", async () => { + const { namespace, keys, expirations } = countingNamespace(); + const gone = `${KV_FORGET_LEDGER_ROOT}t/1:e-1`; + keys.set(gone, { ...at(1), inValue: true }); + const reading = { ...namespace, get: async () => null } as unknown as KVNamespace; + const expiry = await expireKvForgetLedger({ + namespace: reading, + keepForMs: 35 * DAY, + now: NOW, + }); + expect(expiry.expire.map(({ key }) => key)).toEqual([gone]); + expect(expirations.get(gone)).toBeUndefined(); + }); + it("lists every page of a namespace with more keys than a page", async () => { const { namespace, keys, pages } = countingNamespace(); for (let index = 0; index < 1_001; index += 1) { diff --git a/packages/cloudflare/test/read-model-transaction.test.ts b/packages/cloudflare/test/read-model-transaction.test.ts index 70268cef..3bb6ac5f 100644 --- a/packages/cloudflare/test/read-model-transaction.test.ts +++ b/packages/cloudflare/test/read-model-transaction.test.ts @@ -1,6 +1,6 @@ import { runInDurableObject } from "cloudflare:test"; import { env } from "cloudflare:workers"; -import { fieldBuilder as f, silentLogger } from "@bounda-dev/core"; +import { DEFAULT_PERSONAL_CATEGORY, fieldBuilder as f, silentLogger } from "@bounda-dev/core"; import type { AppendArgs, Storage, StorageTransaction } from "@bounda-dev/core/adapter"; import { contractFields, pendingEvent } from "@bounda-dev/core/adapter/testing"; import { describe, expect, it } from "vitest"; @@ -252,7 +252,7 @@ describe("appends in a Durable Object", () => { storage: state.storage, options: {}, }).createStorage(); - await storage.personalKeys.add("c-1", key); + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, key); let release = (): void => {}; const held = new Promise((resolve) => { release = resolve; @@ -277,9 +277,12 @@ describe("appends in a Durable Object", () => { }); it("refuses an append that a transaction forgetting its person held up", async () => { - const { refusal, position } = await heldUp((tx) => tx.personalKeys.remove("c-1"), { - persons: [{ person: "c-1", fingerprint: key.fingerprint }], - }); + const { refusal, position } = await heldUp( + (tx) => tx.personalKeys.remove("c-1", key.fingerprint), + { + persons: [{ person: "c-1", fingerprint: key.fingerprint }], + }, + ); expect(refusal).toMatchObject({ code: "CONCURRENCY_CONFLICT", personId: "c-1" }); expect(position).toBe(0); }); diff --git a/packages/cloudflare/test/storage-contracts.test.ts b/packages/cloudflare/test/storage-contracts.test.ts index dc98e86f..a9aaaac1 100644 --- a/packages/cloudflare/test/storage-contracts.test.ts +++ b/packages/cloudflare/test/storage-contracts.test.ts @@ -62,7 +62,7 @@ describe("the Durable Object adapter", () => { idempotencyStoreContract({ create: freshStorage }); snapshotStoreContract({ create: freshStorage }); storageTransactionContract({ create: freshStorage }); - personalKeyStoreContract({ create: async () => (await freshStorage()).personalKeys }); + personalKeyStoreContract({ create: async () => freshStorage() }); personStoreContract({ create: async () => (await freshStorage()).persons }); inspectionContract({ create: freshAdapter }); subscriberFailureStoreContract({ create: async () => (await freshStorage()).subscriberFailures }); diff --git a/packages/core/src/adapter/deferred-writes.test.ts b/packages/core/src/adapter/deferred-writes.test.ts index d8f1a891..e334c090 100644 --- a/packages/core/src/adapter/deferred-writes.test.ts +++ b/packages/core/src/adapter/deferred-writes.test.ts @@ -24,7 +24,7 @@ const record = (key: string, persons: readonly KeyedPerson[]) => ({ describe("deferWrites", () => { it("forgets the idempotency records it holds back for a person, and forgets them on flush", async () => { const storage = await memory().createStorage(); - await storage.personalKeys.add("c-1", stored("1")); + await storage.personalKeys.add("c-1", "default", stored("1")); await storage.idempotencyStore.put(record("live", [held("c-1", "1")])); const { stores, flush } = deferWrites(storage); await stores.idempotencyStore.put(record("held", [held("c-1", "1")])); @@ -41,6 +41,48 @@ describe("deferWrites", () => { expect(await storage.idempotencyStore.get({ key: "other", now })).not.toBeNull(); }); + it("forgets and withdraws only what names a person under a key outside the ones kept", async () => { + const storage = await memory().createStorage(); + await storage.personalKeys.add("c-1", "default", stored("1")); + await storage.personalKeys.add("c-1", "profile", stored("2")); + const { stores, flush, withdrawScheduled, withdrawLetters } = deferWrites(storage); + for (const [key, fingerprint] of [ + ["kept", "1"], + ["gone", "2"], + ["clear", null], + ] as const) { + await stores.idempotencyStore.put(record(key, [held("c-1", fingerprint)])); + await stores.scheduler.schedule({ + dedupeKey: key, + command: testCommand("c-1"), + executeAt: now, + context: testContext, + persons: [held("c-1", fingerprint)], + }); + await stores.deadLetterStore.add({ + ...testDeadLetter(key), + kind: "scheduled", + persons: [held("c-1", fingerprint)], + }); + } + await stores.idempotencyStore.forget("c-1", ["f-1"]); + expect(withdrawScheduled("c-1", ["f-1"]).map(({ dedupeKey }) => dedupeKey)).toEqual([ + "gone", + "clear", + ]); + expect(withdrawLetters("c-1", ["f-1"]).map(({ id }) => id)).toEqual(["gone", "clear"]); + await flush(storage); + const kept = async (key: string) => + [ + (await storage.idempotencyStore.get({ key, now })) !== null, + (await storage.scheduler.list()).some(({ dedupeKey }) => dedupeKey === key), + (await storage.deadLetterStore.list()).some(({ id }) => id === key), + ] as const; + expect(await kept("kept")).toEqual([true, true, true]); + expect(await kept("gone")).toEqual([false, false, false]); + expect(await kept("clear")).toEqual([false, false, false]); + }); + it("lists what names a person from the live stores, not from the writes held back", async () => { const storage = await memory().createStorage(); const persons = [held("c-1", "1")]; diff --git a/packages/core/src/adapter/deferred-writes.ts b/packages/core/src/adapter/deferred-writes.ts index 14b60977..af12471c 100644 --- a/packages/core/src/adapter/deferred-writes.ts +++ b/packages/core/src/adapter/deferred-writes.ts @@ -42,15 +42,15 @@ export interface DeferredWrites { */ pending(): boolean; /** - * Withdraws the commands scheduled so far, and not flushed, whose `persons` names `person`, and - * resolves to them as the scheduler would hold them: what a `forget` that runs before the flush + * Withdraws the commands scheduled so far, and not flushed, whose `persons` names `person` as + * `namesPerson` tells with `keep`, and resolves to them as the scheduler would hold them: what a `forget` that runs before the flush * must find too, as it does in a store whose transaction sees its own writes. */ - withdrawScheduled(person: string): readonly ScheduledCommand[]; + withdrawScheduled(person: string, keep?: readonly string[]): readonly ScheduledCommand[]; /** * The same for the dead letters added so far. */ - withdrawLetters(person: string): readonly StoredDeadLetter[]; + withdrawLetters(person: string, keep?: readonly string[]): readonly StoredDeadLetter[]; } export interface DeferWritesFunction { @@ -136,11 +136,11 @@ export const deferWrites: DeferWritesFunction = ({ recorded.push((target) => target.idempotencyStore.put(args)); }, purge: idempotencyStore.purge, - forget: async (person) => { + forget: async (person, keep) => { for (const [key, record] of records) { - if (record.persons.some((held) => held.person === person)) records.delete(key); + if (namesPerson(record, person, keep)) records.delete(key); } - recorded.push((target) => target.idempotencyStore.forget(person)); + recorded.push((target) => target.idempotencyStore.forget(person, keep)); }, }, }, @@ -148,8 +148,8 @@ export const deferWrites: DeferWritesFunction = ({ for (const write of recorded) await write(target); }, pending: () => recorded.length > 0, - withdrawScheduled: (person) => { - const named = [...scheduled.values()].filter((args) => namesPerson(args, person)); + withdrawScheduled: (person, keep) => { + const named = [...scheduled.values()].filter((args) => namesPerson(args, person, keep)); for (const { dedupeKey } of named) { scheduled.delete(dedupeKey); recorded.push((target) => target.scheduler.cancel(dedupeKey)); @@ -163,8 +163,8 @@ export const deferWrites: DeferWritesFunction = ({ persons: persons ?? [], })); }, - withdrawLetters: (person) => { - const named = [...letters.values()].filter((letter) => namesPerson(letter, person)); + withdrawLetters: (person, keep) => { + const named = [...letters.values()].filter((letter) => namesPerson(letter, person, keep)); for (const { id } of named) { letters.delete(id); recorded.push((target) => target.deadLetterStore.remove(id)); diff --git a/packages/core/src/adapter/index.ts b/packages/core/src/adapter/index.ts index db2d1f48..786b09c4 100644 --- a/packages/core/src/adapter/index.ts +++ b/packages/core/src/adapter/index.ts @@ -18,7 +18,7 @@ export { isAdapter } from "./adapter.ts"; export type { AdapterDefinition } from "./adapter-definition.ts"; export type { RebuildFencing, RebuildFencingFunction } from "./rebuild-fencing.ts"; export { rebuildFencing } from "./rebuild-fencing.ts"; -export type { CheckAppendArgs, CheckAppendFunction } from "./storage/append-checks.ts"; +export type { CheckAppendArgs, CheckAppendFunction, KeyState } from "./storage/append-checks.ts"; export { checkAppend, ForgottenSinceLoadError } from "./storage/append-checks.ts"; export type { Checkpoint, CheckpointStore } from "./storage/checkpoint-store.ts"; export type { @@ -83,16 +83,22 @@ export type { StoredPerson, } from "./storage/person-store.ts"; export type { + DueKey, + HeldPersonalKey, KeyedPerson, + ListDueArgs, ListWrappedUnderArgs, + NextDueHoldEndArgs, NextKeyHoldEndArgs, PersonalEnvelope, PersonalKeyRow, PersonalKeyStore, + RetainKeyArgs, RewrappedKey, SealedValue, StoredPersonalKey, } from "./storage/personal-key-store.ts"; +export { DEFAULT_PERSONAL_CATEGORY } from "./storage/personal-key-store.ts"; export type { ClaimDueArgs, ClaimedCommand, diff --git a/packages/core/src/adapter/sql/append-checks.test.ts b/packages/core/src/adapter/sql/append-checks.test.ts index bf1caf13..88ae256a 100644 --- a/packages/core/src/adapter/sql/append-checks.test.ts +++ b/packages/core/src/adapter/sql/append-checks.test.ts @@ -45,8 +45,8 @@ describe("checkSqlAppend", () => { it("reads the key rows of the persons named with a key and the rows of those touched, each once, locked when asked", async () => { const { tx, statements } = recording({ keys: [ - { person: "c-1", fingerprint: "f-1" }, - { person: "c-2", fingerprint: "f-2" }, + { person: "c-1", fingerprint: "f-1", retained_since: null }, + { person: "c-2", fingerprint: "f-2", retained_since: null }, ], persons: [{ person: "c-3", restricted_since: null }], }); @@ -67,7 +67,7 @@ describe("checkSqlAppend", () => { }); expect(statements).toEqual([ [ - `SELECT "person", "fingerprint" FROM keys WHERE "person" IN ($1, $2) FOR SHARE`, + `SELECT "person", "fingerprint", "retained_since" FROM keys WHERE "person" IN ($1, $2) FOR SHARE`, ["c-1", "c-2"], ], [ @@ -79,7 +79,7 @@ describe("checkSqlAppend", () => { it("refuses on what it read: a restricted person, or a key no longer held", async () => { const { tx, statements } = recording({ - keys: [{ person: "c-1", fingerprint: "f-new" }], + keys: [{ person: "c-1", fingerprint: "f-new", retained_since: null }], persons: [{ person: "c-2", restricted_since: "2026-01-01T00:00:00.000Z" }], }); await expect( @@ -110,4 +110,25 @@ describe("checkSqlAppend", () => { `SELECT "person", "restricted_since" FROM persons WHERE "person" IN ($1)`, ); }); + + it("refuses a key kept past a forgetting where it was read in force, and lets it through where it was read retained", async () => { + const { tx } = recording({ + keys: [{ person: "c-1", fingerprint: "f-1", retained_since: "2026-01-01T00:00:00.000Z" }], + }); + const check = (retained: boolean) => + checkSqlAppend({ + tx, + dialect: postgresqlDialect, + tables, + args: append({ + persons: [ + retained + ? { person: "c-1", fingerprint: "f-1", retained: true } + : { person: "c-1", fingerprint: "f-1" }, + ], + }), + }); + await expect(check(false)).rejects.toBeInstanceOf(ForgottenSinceLoadError); + await expect(check(true)).resolves.toBeUndefined(); + }); }); diff --git a/packages/core/src/adapter/sql/append-checks.ts b/packages/core/src/adapter/sql/append-checks.ts index 6f66f62d..1b2dee51 100644 --- a/packages/core/src/adapter/sql/append-checks.ts +++ b/packages/core/src/adapter/sql/append-checks.ts @@ -1,5 +1,6 @@ import { appendPersonsOf, checkAppend } from "../storage/append-checks.ts"; import type { AppendArgs } from "../storage/event-store.ts"; +import { keyedPersonId } from "../storage/personal-key-store.ts"; import type { SqlDialect } from "./dialect.ts"; import type { SqlExecutor } from "./sql-table.ts"; @@ -44,15 +45,22 @@ export const checkSqlAppend: CheckSqlAppendFunction = async ({ persons, ); }; - const keys = await read(`"person", "fingerprint"`, tables.personalKeys, keyed); + const keys = await read(`"person", "fingerprint", "retained_since"`, tables.personalKeys, keyed); const rows = await read(`"person", "restricted_since"`, tables.persons, touched); - const fingerprints = new Map(keys.map((row) => [String(row.person), String(row.fingerprint)])); + const states = new Map( + keys.map((row) => [ + keyedPersonId({ person: String(row.person), fingerprint: String(row.fingerprint) }), + row.retained_since === null || row.retained_since === undefined + ? ("held" as const) + : ("retained" as const), + ]), + ); const restricted = new Set( rows.filter((row) => row.restricted_since !== null).map((row) => String(row.person)), ); checkAppend({ args, - fingerprintOf: (person) => fingerprints.get(person) ?? null, + keyOf: (person, fingerprint) => states.get(keyedPersonId({ person, fingerprint })) ?? "gone", restricted: (person) => restricted.has(person), }); }; diff --git a/packages/core/src/adapter/sql/forget-ledger.test.ts b/packages/core/src/adapter/sql/forget-ledger.test.ts index d24c3e45..76c82ff8 100644 --- a/packages/core/src/adapter/sql/forget-ledger.test.ts +++ b/packages/core/src/adapter/sql/forget-ledger.test.ts @@ -3,11 +3,12 @@ import type { ForgetLedgerEntry } from "../storage/forget-ledger.ts"; import { postgresqlDialect, sqliteDialect } from "./dialect.ts"; import { createSqlForgetLedger, + forgetLedgerCategoriesStatement, forgetLedgerReasonStatements, forgetLedgerStatements, } from "./forget-ledger.ts"; -const recording = () => { +const recording = (rows: readonly Record[] = []) => { const statements: (readonly [string, readonly unknown[]])[] = []; const db = { run: async (sql: string, params: readonly unknown[]) => { @@ -15,7 +16,7 @@ const recording = () => { }, all: async (sql: string, params: readonly unknown[]) => { statements.push([sql, params]); - return []; + return rows; }, }; return { db, statements }; @@ -25,6 +26,7 @@ const entry = (index: number): ForgetLedgerEntry => ({ personId: `c-${index}`, eventId: `e-${index}`, forgottenAt: "2026-01-01T00:00:00.000Z", + categories: index % 2 === 0 ? null : ["profile"], reason: "requested", fingerprints: [`k-${index}`], }); @@ -48,7 +50,16 @@ describe("the SQL forget ledger", () => { ]); }); - it("inserts at most 16 entries a statement, each bound to its own numbered parameters", async () => { + it("adds the categories in a migration of their own, again harmlessly on PostgreSQL", () => { + expect(forgetLedgerCategoriesStatement(`"t_forget_ledger"`, postgresqlDialect)).toBe( + `ALTER TABLE "t_forget_ledger" ADD COLUMN IF NOT EXISTS "categories" TEXT`, + ); + expect(forgetLedgerCategoriesStatement(`"t_forget_ledger"`, sqliteDialect)).toBe( + `ALTER TABLE "t_forget_ledger" ADD COLUMN "categories" TEXT`, + ); + }); + + it("inserts at most 14 entries a statement, each bound to its own numbered parameters", async () => { const { db, statements } = recording(); const ledger = createSqlForgetLedger({ db, @@ -58,37 +69,76 @@ describe("the SQL forget ledger", () => { ensureSchema: async () => {}, close: async () => {}, }); - const { fingerprints: _, ...unrecorded } = entry(16); + const { fingerprints: _, ...unrecorded } = entry(14); await ledger.append([ - ...Array.from({ length: 16 }, (_, index) => entry(index)), + ...Array.from({ length: 14 }, (_, index) => entry(index)), { ...unrecorded, reason: "retention" }, ]); expect(statements).toHaveLength(2); const [first, second] = statements; expect(first?.[0]).toContain( - `INSERT INTO "t" ("tenant", "event_id", "person", "forgotten_at", "reason", "fingerprints") VALUES ($1, $2, $3, $4, $5, $6), ($7, $8, $9, $10, $11, $12)`, + `INSERT INTO "t" ("tenant", "event_id", "person", "forgotten_at", "reason", "fingerprints", "categories") VALUES ($1, $2, $3, $4, $5, $6, $7), ($8, $9, $10, $11, $12, $13, $14)`, ); - expect(first?.[0]).toContain("($91, $92, $93, $94, $95, $96) ON CONFLICT"); - expect(first?.[1]).toHaveLength(96); - expect(first?.[1].slice(0, 6)).toEqual([ + expect(first?.[0]).toContain("($92, $93, $94, $95, $96, $97, $98) ON CONFLICT"); + expect(first?.[1]).toHaveLength(98); + expect(first?.[1].slice(0, 14)).toEqual([ "", "e-0", "c-0", "2026-01-01T00:00:00.000Z", "requested", `["k-0"]`, + null, + "", + "e-1", + "c-1", + "2026-01-01T00:00:00.000Z", + "requested", + `["k-1"]`, + `["profile"]`, ]); - expect(second?.[0]).toContain(`VALUES ($1, $2, $3, $4, $5, $6) ON CONFLICT`); + expect(second?.[0]).toContain(`VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT`); expect(second?.[1]).toEqual([ "", - "e-16", - "c-16", + "e-14", + "c-14", "2026-01-01T00:00:00.000Z", "retention", null, + null, ]); }); + it("reads the categories of an entry back, null for a forgetting of the person as a whole", async () => { + const { db } = recording([ + { + event_id: "e-0", + person: "c-0", + forgotten_at: "2026-01-01T00:00:00.000Z", + reason: "requested", + fingerprints: `["k-0"]`, + categories: null, + }, + { + event_id: "e-1", + person: "c-1", + forgotten_at: "2026-01-01T00:00:00.000Z", + reason: "requested", + fingerprints: `["k-1"]`, + categories: `["profile"]`, + }, + ]); + const ledger = createSqlForgetLedger({ + db, + table: `"t"`, + dialect: sqliteDialect, + tenant: undefined, + ensureSchema: async () => {}, + close: async () => {}, + }); + expect(await ledger.read({ limit: 2 })).toEqual([entry(0), entry(1)]); + }); + it("creates its table before the first statement, and again after that failed", async () => { const { db, statements } = recording(); let attempts = 0; diff --git a/packages/core/src/adapter/sql/forget-ledger.ts b/packages/core/src/adapter/sql/forget-ledger.ts index 4079bb3a..2fc0e84e 100644 --- a/packages/core/src/adapter/sql/forget-ledger.ts +++ b/packages/core/src/adapter/sql/forget-ledger.ts @@ -3,7 +3,15 @@ import type { SqlDialect } from "./dialect.ts"; import { quoteIdentifier } from "./identifiers.ts"; import type { SqlExecutor } from "./sql-table.ts"; -const COLUMNS = ["tenant", "event_id", "person", "forgotten_at", "reason", "fingerprints"] as const; +const COLUMNS = [ + "tenant", + "event_id", + "person", + "forgotten_at", + "reason", + "fingerprints", + "categories", +] as const; // Within the hundred parameters a Durable Object's SQLite binds to one statement. const ENTRIES_PER_INSERT = Math.floor(100 / COLUMNS.length); @@ -87,9 +95,26 @@ export interface CreateSqlForgetLedgerFunction { (args: CreateSqlForgetLedgerArgs): ForgetLedger; } +export interface ForgetLedgerCategoriesStatementFunction { + (table: string, dialect: Pick): string; +} + +/** + * The fourth version: the categories each forgetting was asked for, as a JSON array, `NULL` for + * the person as a whole, which every entry written before was. SQLite has no `IF NOT EXISTS` + * there, so its caller runs it only on a table without the column. + */ +export const forgetLedgerCategoriesStatement: ForgetLedgerCategoriesStatementFunction = ( + table, + { name }, +) => + `ALTER TABLE ${table} ADD COLUMN ${name === "postgresql" ? "IF NOT EXISTS " : ""}"categories" TEXT`; + const entryOf = (row: Record): ForgetLedgerEntry => { + const categories: unknown = JSON.parse(String(row.categories ?? null)); const entry: ForgetLedgerEntry = { personId: String(row.person), + categories: Array.isArray(categories) ? categories.map(String) : null, eventId: String(row.event_id), forgottenAt: String(row.forgotten_at), reason: row.reason === "retention" ? "retention" : "requested", @@ -98,7 +123,7 @@ const entryOf = (row: Record): ForgetLedgerEntry => { return Array.isArray(fingerprints) ? { ...entry, fingerprints: fingerprints.map(String) } : entry; }; -const SELECTED = `"event_id", "person", "forgotten_at", "reason", "fingerprints"`; +const SELECTED = `"event_id", "person", "forgotten_at", "reason", "fingerprints", "categories"`; /** * The forget ledger of SQLite and PostgreSQL alike, in a table of its own that holds every @@ -133,13 +158,14 @@ export const createSqlForgetLedger: CreateSqlForgetLedgerFunction = ({ ); await db.run( `INSERT INTO ${table} (${COLUMNS.map((column) => `"${column}"`).join(", ")}) VALUES ${rows.join(", ")} ON CONFLICT ("tenant", "event_id") DO NOTHING`, - chunk.flatMap(({ eventId, personId, forgottenAt, reason, fingerprints }) => [ + chunk.flatMap(({ eventId, personId, forgottenAt, reason, fingerprints, categories }) => [ tenant, eventId, personId, forgottenAt, reason, fingerprints === undefined ? null : JSON.stringify(fingerprints), + categories === null ? null : JSON.stringify(categories), ]), ); } diff --git a/packages/core/src/adapter/sql/index.ts b/packages/core/src/adapter/sql/index.ts index b40d294f..d3bccc6a 100644 --- a/packages/core/src/adapter/sql/index.ts +++ b/packages/core/src/adapter/sql/index.ts @@ -22,6 +22,7 @@ export type { } from "./forget-ledger.ts"; export { createSqlForgetLedger, + forgetLedgerCategoriesStatement, forgetLedgerMarksStatement, forgetLedgerMarksTableFor, forgetLedgerReasonStatements, diff --git a/packages/core/src/adapter/sql/person-store.ts b/packages/core/src/adapter/sql/person-store.ts index da199a40..f0a7bd80 100644 --- a/packages/core/src/adapter/sql/person-store.ts +++ b/packages/core/src/adapter/sql/person-store.ts @@ -16,7 +16,7 @@ export interface CreateSqlPersonStoreFunction { (args: CreateSqlPersonStoreArgs): PersonStore; } -const COLUMNS = `"person", "restricted_since", "last_active_at", "forgotten_at", "held_until", "attempts"`; +const COLUMNS = `"person", "restricted_since", "last_active_at", "position", "forgotten_at", "held_until", "attempts"`; // The predicate of the index the schemas create for `listIdle`, which a query must repeat for the // index to serve it. @@ -31,6 +31,7 @@ const storedPerson = (row: Record): StoredPerson => ({ person: String(row.person), restrictedSince: text(row.restricted_since), lastActiveAt: text(row.last_active_at), + position: Number(row.position), forgottenAt: text(row.forgotten_at), hold: holdOf(row), }); diff --git a/packages/core/src/adapter/sql/personal-key-store.ts b/packages/core/src/adapter/sql/personal-key-store.ts index 6df06240..a309f432 100644 --- a/packages/core/src/adapter/sql/personal-key-store.ts +++ b/packages/core/src/adapter/sql/personal-key-store.ts @@ -1,4 +1,9 @@ -import type { PersonalKeyStore, StoredPersonalKey } from "../storage/personal-key-store.ts"; +import type { + DueKey, + HeldPersonalKey, + PersonalKeyStore, + StoredPersonalKey, +} from "../storage/personal-key-store.ts"; import type { SqlDialect } from "./dialect.ts"; import { holdEndOf, holdOf, releasedBy as released } from "./hold.ts"; import type { SqlExecutor } from "./sql-table.ts"; @@ -6,9 +11,10 @@ import type { SqlExecutor } from "./sql-table.ts"; export interface CreateSqlPersonalKeyStoreArgs { readonly db: SqlExecutor; /** - * Already quoted. + * Both already quoted: the keys, and the persons whose restriction `listDue` reads. */ readonly table: string; + readonly persons: string; readonly dialect: Pick; } @@ -16,20 +22,54 @@ export interface CreateSqlPersonalKeyStoreFunction { (args: CreateSqlPersonalKeyStoreArgs): PersonalKeyStore; } +const text = (value: unknown): string | null => (value === null ? null : String(value)); + +const keyOf = (row: Record): StoredPersonalKey => ({ + key: String(row.key), + fingerprint: String(row.fingerprint), + kid: row.kid === null ? null : String(row.kid), +}); + +const heldOf = (row: Record): HeldPersonalKey => ({ + category: String(row.category), + key: keyOf(row), + retainedSince: text(row.retained_since), + lastActiveAt: text(row.last_active_at), +}); + +const HELD = `"category", "key", "fingerprint", "kid", "retained_since", "last_active_at"`; + +// The predicate of the unique index that keeps one key in force per person and category, which an +// `ON CONFLICT` on it repeats. +const IN_FORCE = `"retained_since" IS NULL`; + +// The predicate of the index `listDue` reads. +const ACTIVE = `"last_active_at" IS NOT NULL`; + +// The predicate of the index `listRetained` reads. +const RETAINED = `"retained_since" IS NOT NULL`; + +const DUE = `"person", "category", "fingerprint", "last_active_at", "retained_since", "forget_held_until", "forget_attempts"`; + +const dueOf = (row: Record): DueKey => ({ + person: String(row.person), + category: String(row.category), + fingerprint: String(row.fingerprint), + lastActiveAt: String(row.last_active_at), + retainedSince: text(row.retained_since), + hold: holdOf({ held_until: row.forget_held_until, attempts: row.forget_attempts }), +}); + /** * The personal key store of SQLite and PostgreSQL alike, on the SQL both speak. */ export const createSqlPersonalKeyStore: CreateSqlPersonalKeyStoreFunction = ({ db, table, + persons, dialect, }) => { const p = dialect.placeholder; - const keyOf = (row: Record): StoredPersonalKey => ({ - key: String(row.key), - fingerprint: String(row.fingerprint), - kid: row.kid === null ? null : String(row.kid), - }); // One equality per id, so the index on `kid` answers each. const wrappedUnder = ( kids: readonly (string | null)[], @@ -43,26 +83,126 @@ export const createSqlPersonalKeyStore: CreateSqlPersonalKeyStoreFunction = ({ }; // The row as `from` was read: `rewrap` and `hold` change it only while it holds that key. const holding = `"person" = ${p(3)} AND "key" = ${p(4)} AND "fingerprint" = ${p(5)}`; - const get = async (person: string): Promise => { + // A key whose person is restricted is left alone, so a list never fills up with such keys. + const unrestricted = `NOT EXISTS (SELECT 1 FROM ${persons} AS "r" WHERE "r"."person" = ${table}."person" AND "r"."restricted_since" IS NOT NULL)`; + const dueReleasedBy = (placeholder: string): string => + released(placeholder).replaceAll('"held_until"', '"forget_held_until"'); + const get = async (person: string, category: string): Promise => { const [row] = await db.all( - `SELECT "key", "fingerprint", "kid" FROM ${table} WHERE "person" = ${p(1)}`, - [person], + `SELECT "key", "fingerprint", "kid" FROM ${table} WHERE "person" = ${p(1)} AND "category" = ${p(2)} AND ${IN_FORCE}`, + [person, category], ); return row === undefined ? null : keyOf(row); }; return { get, - // Whichever writer inserted first, every one answers with the key in force. - add: async (person, key) => { + find: async (person, fingerprint) => { + const [row] = await db.all( + `SELECT ${HELD} FROM ${table} WHERE "person" = ${p(1)} AND "fingerprint" = ${p(2)}`, + [person, fingerprint], + ); + return row === undefined ? null : heldOf(row); + }, + list: async (person) => + ( + await db.all( + `SELECT ${HELD} FROM ${table} WHERE "person" = ${p(1)} ORDER BY "fingerprint"`, + [person], + ) + ).map(heldOf), + // Whichever writer inserted first, every one answers with the key in force. A new key starts + // from the person's last activity recorded since their last forgetting as a whole, so a key + // made for an inactive person, as the backfill makes, still comes due. + add: async (person, category, key) => { await db.run( - `INSERT INTO ${table} ("person", "key", "fingerprint", "kid") VALUES (${p(1)}, ${p(2)}, ${p(3)}, ${p(4)}) ON CONFLICT ("person") DO NOTHING`, - [person, key.key, key.fingerprint, key.kid], + `INSERT INTO ${table} ("person", "category", "key", "fingerprint", "kid", "last_active_at") VALUES (${p(1)}, ${p(2)}, ${p(3)}, ${p(4)}, ${p(5)}, (SELECT "last_active_at" FROM ${persons} WHERE "person" = ${p(6)} AND "forgotten_at" IS NULL)) ON CONFLICT ("person", "category") WHERE ${IN_FORCE} DO NOTHING`, + [person, category, key.key, key.fingerprint, key.kid, person], + ); + return (await get(person, category)) ?? key; + }, + remove: async (person, fingerprint) => + ( + await db.all( + `DELETE FROM ${table} WHERE "person" = ${p(1)} AND "fingerprint" = ${p(2)} RETURNING "person"`, + [person, fingerprint], + ) + ).length === 1, + retain: async (person, fingerprint, { since, lastActiveAt }) => + ( + await db.all( + `UPDATE ${table} SET "retained_since" = ${p(1)}, "last_active_at" = ${p(2)}, "forget_held_until" = NULL, "forget_attempts" = 0 WHERE "person" = ${p(3)} AND "fingerprint" = ${p(4)} AND ${IN_FORCE} RETURNING "person"`, + [since, lastActiveAt, person, fingerprint], + ) + ).length === 1, + touch: async (person, lastActiveAt) => { + await db.run( + `UPDATE ${table} SET "last_active_at" = ${p(1)}, "forget_held_until" = NULL, "forget_attempts" = 0 WHERE "person" = ${p(2)} AND ${IN_FORCE} AND ("last_active_at" IS NULL OR "last_active_at" < ${p(3)})`, + [lastActiveAt, person, lastActiveAt], + ); + }, + listDue: async ({ category, activeBefore, releasedBy, limit }) => { + const conditions = [`"category" = ${p(1)}`, ACTIVE, unrestricted]; + const values: unknown[] = [category]; + if (activeBefore !== undefined) { + values.push(activeBefore.toISOString()); + conditions.push(`"last_active_at" <= ${p(values.length)}`); + } + if (releasedBy !== undefined) { + values.push(releasedBy.toISOString()); + conditions.push(dueReleasedBy(p(values.length))); + } + values.push(limit); + const rows = await db.all( + `SELECT ${DUE} FROM ${table} WHERE ${conditions.join(" AND ")} ORDER BY "last_active_at", "person", "fingerprint" LIMIT ${p(values.length)}`, + values, + ); + return rows.map(dueOf); + }, + holdDue: async (person, fingerprint, lastActiveAt, { until, attempts }) => { + await db.run( + `UPDATE ${table} SET "forget_held_until" = ${p(1)}, "forget_attempts" = ${p(2)} WHERE "person" = ${p(3)} AND "fingerprint" = ${p(4)} AND "last_active_at" = ${p(5)}`, + [until, attempts, person, fingerprint, lastActiveAt], + ); + }, + nextDueHoldEnd: async ({ category, after }) => + holdEndOf( + await db.all( + `SELECT MIN("forget_held_until") AS "until" FROM ${table} WHERE "category" = ${p(1)} AND ${ACTIVE} AND ${unrestricted} AND "forget_held_until" > ${p(2)}`, + [category, after.toISOString()], + ), + ), + // A key a forgetting retained has its activity frozen, so `last_active_at` is never null. + listRetained: async ({ except, releasedBy, limit }) => { + const conditions = [RETAINED, unrestricted]; + const values: unknown[] = []; + if (except.length > 0) { + conditions.push(`"category" NOT IN (${except.map((_, index) => p(index + 1)).join(", ")})`); + values.push(...except); + } + if (releasedBy !== undefined) { + values.push(releasedBy.toISOString()); + conditions.push(dueReleasedBy(p(values.length))); + } + values.push(limit); + const rows = await db.all( + `SELECT ${DUE} FROM ${table} WHERE ${conditions.join(" AND ")} ORDER BY "retained_since", "person", "fingerprint" LIMIT ${p(values.length)}`, + values, + ); + return rows.map(dueOf); + }, + nextRetainedHoldEnd: async ({ except, after }) => { + const values: unknown[] = [...except, after.toISOString()]; + const outside = + except.length === 0 + ? "" + : ` AND "category" NOT IN (${except.map((_, index) => p(index + 1)).join(", ")})`; + return holdEndOf( + await db.all( + `SELECT MIN("forget_held_until") AS "until" FROM ${table} WHERE ${RETAINED}${outside} AND ${unrestricted} AND "forget_held_until" > ${p(values.length)}`, + values, + ), ); - return (await get(person)) ?? key; }, - remove: async (person) => - (await db.all(`DELETE FROM ${table} WHERE "person" = ${p(1)} RETURNING "person"`, [person])) - .length === 1, listWrappedUnder: async ({ kids, releasedBy, limit }) => { const under = wrappedUnder(kids); if (under === null) return []; @@ -74,11 +214,12 @@ export const createSqlPersonalKeyStore: CreateSqlPersonalKeyStoreFunction = ({ } values.push(limit); const rows = await db.all( - `SELECT "person", "key", "fingerprint", "kid", "held_until", "attempts" FROM ${table} WHERE ${conditions.join(" AND ")} ORDER BY "person" LIMIT ${p(values.length)}`, + `SELECT "person", "category", "key", "fingerprint", "kid", "held_until", "attempts" FROM ${table} WHERE ${conditions.join(" AND ")} ORDER BY "person", "fingerprint" LIMIT ${p(values.length)}`, values, ); return rows.map((row) => ({ person: String(row.person), + category: String(row.category), key: keyOf(row), hold: holdOf(row), })); @@ -123,5 +264,36 @@ export const createSqlPersonalKeyStore: CreateSqlPersonalKeyStoreFunction = ({ [], ) ).length, + countDue: async ({ category, activeBefore }) => { + const conditions = [`"category" = ${p(1)}`, ACTIVE, unrestricted]; + const values: unknown[] = [category]; + if (activeBefore !== undefined) { + values.push(activeBefore.toISOString()); + conditions.push(`"last_active_at" <= ${p(values.length)}`); + } + const [row] = await db.all( + `SELECT count(*) AS "count" FROM ${table} WHERE ${conditions.join(" AND ")}`, + values, + ); + return Number(row?.count ?? 0); + }, + countRetained: async ({ except }) => { + const outside = + except.length === 0 + ? "" + : ` AND "category" NOT IN (${except.map((_, index) => p(index + 1)).join(", ")})`; + const [row] = await db.all( + `SELECT count(*) AS "count" FROM ${table} WHERE ${RETAINED}${outside} AND ${unrestricted}`, + [...except], + ); + return Number(row?.count ?? 0); + }, + releaseDueHolds: async () => + ( + await db.all( + `UPDATE ${table} SET "forget_held_until" = NULL, "forget_attempts" = 0 WHERE "forget_held_until" IS NOT NULL RETURNING "person"`, + [], + ) + ).length, }; }; diff --git a/packages/core/src/adapter/sqlite/adapter.ts b/packages/core/src/adapter/sqlite/adapter.ts index 8f6014b0..e21d66e3 100644 --- a/packages/core/src/adapter/sqlite/adapter.ts +++ b/packages/core/src/adapter/sqlite/adapter.ts @@ -178,6 +178,7 @@ export const createSqliteAdapter: CreateSqliteAdapterFunction = < personalKeys: createSqlPersonalKeyStore({ db: database, table: tables.personalKeys, + persons: tables.persons, dialect: sqliteDialect, }), persons: createSqlPersonStore({ diff --git a/packages/core/src/adapter/sqlite/dead-letter-store.ts b/packages/core/src/adapter/sqlite/dead-letter-store.ts index e98a44b1..56a06fb8 100644 --- a/packages/core/src/adapter/sqlite/dead-letter-store.ts +++ b/packages/core/src/adapter/sqlite/dead-letter-store.ts @@ -10,7 +10,7 @@ import type { } from "../index.ts"; import type { SqlDatabase } from "../sql/database.ts"; import { byFirstFailure, decodeErrorText, encodeErrorText } from "../sql/dead-letters.ts"; -import { naming } from "./held-keys.ts"; +import { naming, outsideKept } from "./held-keys.ts"; import { PERSON_INDEXED } from "./schema.ts"; export interface CreateSqliteDeadLetterStoreArgs { @@ -170,8 +170,13 @@ export const createSqliteDeadLetterStore: CreateSqliteDeadLetterStoreFunction = [person, person], ) ).map(toLetter), - forget: async (person) => - (await db.all(`DELETE FROM ${table} WHERE ${names} RETURNING ${COLUMNS}`, [person])) + forget: async (person, keep = []) => + ( + await db.all( + `DELETE FROM ${table} WHERE ${names} AND ${outsideKept(table)} RETURNING ${COLUMNS}`, + [person, person, JSON.stringify(keep)], + ) + ) .map(toLetter) .sort(byFirstFailure), firstResolvedAt: async () => { diff --git a/packages/core/src/adapter/sqlite/held-keys.ts b/packages/core/src/adapter/sqlite/held-keys.ts index f65256d5..f2e0593a 100644 --- a/packages/core/src/adapter/sqlite/held-keys.ts +++ b/packages/core/src/adapter/sqlite/held-keys.ts @@ -6,12 +6,24 @@ export interface HeldKeysFunction { /** * A condition true when every person in the JSON array of the first parameter named with a - * fingerprint holds, in `personalKeys`, the key of that fingerprint; the second parameter is the - * same array. A person named with `null` was read in the clear and needs no key. A person named - * under two keys never holds both. + * fingerprint holds, in `personalKeys`, the key of that fingerprint in force; the second parameter + * is the same array. A person named with `null` was read in the clear and needs no key. Each entry + * names one key, which a person holds at most once. */ export const heldKeys: HeldKeysFunction = (personalKeys) => - `(SELECT count(*) FROM ${personalKeys} AS "k" JOIN json_each(?) AS "p" ON "k"."person" = json_extract("p"."value", '$.person') AND "k"."fingerprint" = json_extract("p"."value", '$.fingerprint')) = (SELECT count(*) FROM json_each(?) WHERE json_extract("value", '$.fingerprint') IS NOT NULL)`; + `(SELECT count(*) FROM ${personalKeys} AS "k" JOIN json_each(?) AS "p" ON "k"."person" = json_extract("p"."value", '$.person') AND "k"."fingerprint" = json_extract("p"."value", '$.fingerprint') WHERE "k"."retained_since" IS NULL) = (SELECT count(*) FROM json_each(?) WHERE json_extract("value", '$.fingerprint') IS NOT NULL)`; + +export interface OutsideKeptFunction { + (row: string): string; +} + +/** + * A condition true for a row of the table `row` whose `persons` names the person of the first + * parameter under a key outside the JSON array of fingerprints of the second, or read in the + * clear. + */ +export const outsideKept: OutsideKeptFunction = (row) => + `EXISTS (SELECT 1 FROM json_each(${row}."persons") AS "n" WHERE json_extract("n"."value", '$.person') = ? AND (json_extract("n"."value", '$.fingerprint') IS NULL OR json_extract("n"."value", '$.fingerprint') NOT IN (SELECT "value" FROM json_each(?))))`; export interface NamingArgs { readonly personRows: string; diff --git a/packages/core/src/adapter/sqlite/idempotency-store.ts b/packages/core/src/adapter/sqlite/idempotency-store.ts index 54c9342f..b09a12eb 100644 --- a/packages/core/src/adapter/sqlite/idempotency-store.ts +++ b/packages/core/src/adapter/sqlite/idempotency-store.ts @@ -6,7 +6,7 @@ import type { KeyedPerson, } from "../index.ts"; import type { SqlDatabase } from "../sql/database.ts"; -import { heldKeys, naming } from "./held-keys.ts"; +import { heldKeys, naming, outsideKept } from "./held-keys.ts"; import { PERSON_INDEXED } from "./schema.ts"; export interface CreateSqliteIdempotencyStoreArgs { @@ -81,10 +81,10 @@ export const createSqliteIdempotencyStore: CreateSqliteIdempotencyStoreFunction ]); return rows.length; }, - forget: (person) => + forget: (person, keep = []) => db.run( - `DELETE FROM ${table} WHERE ${naming({ personRows, indexed: PERSON_INDEXED.idempotencyKeys })}`, - [person], + `DELETE FROM ${table} WHERE ${naming({ personRows, indexed: PERSON_INDEXED.idempotencyKeys })} AND ${outsideKept(table)}`, + [person, person, JSON.stringify(keep)], ), }; }; diff --git a/packages/core/src/adapter/sqlite/migrations.ts b/packages/core/src/adapter/sqlite/migrations.ts index 81b65840..17be7484 100644 --- a/packages/core/src/adapter/sqlite/migrations.ts +++ b/packages/core/src/adapter/sqlite/migrations.ts @@ -3,6 +3,7 @@ import type { SqlDatabase, SqlTransaction } from "../sql/database.ts"; import { resolvedAtOfEarlierLetters } from "../sql/dead-letters.ts"; import { sqliteDialect } from "../sql/dialect.ts"; import { + forgetLedgerCategoriesStatement, forgetLedgerMarksStatement, forgetLedgerReasonStatements, forgetLedgerStatements, @@ -117,6 +118,55 @@ const adoptStorageTables = async (tx: SqlTransaction, tables: StorageTables): Pr for (const statement of personRowsBackfillStatements(tables)) await tx.run(statement, []); }; +const keysIndex = (tables: StorageTables, suffix: string): string => + quoteIdentifier(`${tables.personalKeys.slice(1, -1)}_${suffix}_idx`); + +/** + * The third migration: a key per person and category instead of one per person, each named by + * its fingerprint, with the person's last activity on it and whether a forgetting kept it under a + * legal obligation. Every key there was is the person's key in the default category, last active + * when the person was, unless they were forgotten after that, when the key came later. Every + * forgetting recorded before forgot the person as a whole, in the default category. + */ +const adoptCategories = async (tx: SqlTransaction, tables: StorageTables): Promise => { + const keys = tables.personalKeys; + const next = quoteIdentifier(`${keys.slice(1, -1)}_next`); + const forgettings = `UPDATE ${tables.events} SET "payload" = json_set("payload", '$.categories', json('["default"]'), '$.retained', json('[]'), '$.whole', json('true')) WHERE "aggregate_type" = 'bounda' AND "type" = 'PersonalDataForgotten' AND json_type("payload", '$.categories') IS NULL`; + // A database migrated before, whose versions were lost, has the keys as this leaves them. + const rebuilt = (await columnsOf(tx, keys)).has("category"); + const statements = [ + `CREATE TABLE ${next} ( + "person" TEXT NOT NULL, + "category" TEXT NOT NULL, + "fingerprint" TEXT NOT NULL, + "key" TEXT NOT NULL, + "kid" TEXT, + "held_until" TEXT, + "attempts" INTEGER NOT NULL DEFAULT 0, + "last_active_at" TEXT, + "retained_since" TEXT, + "forget_held_until" TEXT, + "forget_attempts" INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY ("person", "fingerprint") + )`, + `INSERT INTO ${next} ("person", "category", "fingerprint", "key", "kid", "held_until", "attempts", "last_active_at") SELECT "k"."person", 'default', "k"."fingerprint", "k"."key", "k"."kid", "k"."held_until", "k"."attempts", (SELECT "p"."last_active_at" FROM ${tables.persons} AS "p" WHERE "p"."person" = "k"."person" AND "p"."forgotten_at" IS NULL) FROM ${keys} AS "k"`, + `DROP TABLE ${keys}`, + `ALTER TABLE ${next} RENAME TO ${keys}`, + // One key in force per person and category, which `add` resolves its race on. + `CREATE UNIQUE INDEX ${keysIndex(tables, "in_force")} ON ${keys} ("person", "category") WHERE "retained_since" IS NULL`, + // What the rotation of the wrapping key reads: the keys still under an old one. + `CREATE INDEX ${keysIndex(tables, "kid")} ON ${keys} ("kid", "person")`, + // When the rotation is next due while every key it has left is held. + `CREATE INDEX ${keysIndex(tables, "held")} ON ${keys} ("held_until") WHERE "held_until" IS NOT NULL`, + // What a category's retention reads, least recently active first. + `CREATE INDEX ${keysIndex(tables, "due")} ON ${keys} ("category", "last_active_at") WHERE "last_active_at" IS NOT NULL`, + // The keys a forgetting kept, which no legal obligation may keep any more. + `CREATE INDEX ${keysIndex(tables, "retained")} ON ${keys} ("retained_since") WHERE "retained_since" IS NOT NULL`, + ]; + for (const statement of rebuilt ? [] : statements) await tx.run(statement, []); + await tx.run(forgettings, []); +}; + export interface SqliteStorageScopeFunction { (tables: StorageTables): SchemaScope; } @@ -129,6 +179,7 @@ export const sqliteStorageScope: SqliteStorageScopeFunction = (tables) => ({ version: 2, migrate: (tx) => tx.run(subscriberFailuresTableStatement(tables.subscriberFailures), []), }, + { version: 3, migrate: (tx) => adoptCategories(tx, tables) }, ], }); @@ -163,6 +214,13 @@ export const sqliteForgetLedgerScope: SqliteForgetLedgerScopeFunction = (table) for (const statement of forgetLedgerReasonStatements(table)) await tx.run(statement, []); }, }, + { + version: 4, + migrate: async (tx) => { + if ((await columnsOf(tx, table)).has("categories")) return; + await tx.run(forgetLedgerCategoriesStatement(table, sqliteDialect), []); + }, + }, ], }); diff --git a/packages/core/src/adapter/sqlite/scheduler.ts b/packages/core/src/adapter/sqlite/scheduler.ts index 490b4eda..8206cb18 100644 --- a/packages/core/src/adapter/sqlite/scheduler.ts +++ b/packages/core/src/adapter/sqlite/scheduler.ts @@ -9,7 +9,7 @@ import type { } from "../index.ts"; import type { SqlDatabase } from "../sql/database.ts"; import { byExecuteAt, earliestDue } from "../sql/index.ts"; -import { naming } from "./held-keys.ts"; +import { naming, outsideKept } from "./held-keys.ts"; import { PERSON_INDEXED } from "./schema.ts"; export interface CreateSqliteSchedulerArgs { @@ -186,8 +186,13 @@ export const createSqliteScheduler: CreateSqliteSchedulerFunction = ({ db, table [person], ) ).map(toScheduled), - forget: async (person) => - (await db.all(`DELETE FROM ${table} WHERE ${names} RETURNING ${COLUMNS}`, [person])) + forget: async (person, keep = []) => + ( + await db.all( + `DELETE FROM ${table} WHERE ${names} AND ${outsideKept(table)} RETURNING ${COLUMNS}`, + [person, person, JSON.stringify(keep)], + ) + ) .map(toScheduled) .sort(byExecuteAt), }; diff --git a/packages/core/src/adapter/sqlite/snapshot-store.ts b/packages/core/src/adapter/sqlite/snapshot-store.ts index a96ffe82..16e7b586 100644 --- a/packages/core/src/adapter/sqlite/snapshot-store.ts +++ b/packages/core/src/adapter/sqlite/snapshot-store.ts @@ -1,6 +1,6 @@ import type { KeyedPerson, Snapshot, SnapshotStore } from "../index.ts"; import type { SqlExecutor } from "../sql/sql-table.ts"; -import { heldKeys, naming } from "./held-keys.ts"; +import { heldKeys, naming, outsideKept } from "./held-keys.ts"; import { PERSON_INDEXED } from "./schema.ts"; export interface CreateSqliteSnapshotStoreArgs { @@ -58,9 +58,9 @@ export const createSqliteSnapshotStore: CreateSqliteSnapshotStoreFunction = ({ aggregateType, aggregateId, ]), - forget: (person) => + forget: (person, keep = []) => db.run( - `DELETE FROM ${table} WHERE ${naming({ personRows, indexed: PERSON_INDEXED.snapshots })}`, - [person], + `DELETE FROM ${table} WHERE ${naming({ personRows, indexed: PERSON_INDEXED.snapshots })} AND ${outsideKept(table)}`, + [person, person, JSON.stringify(keep)], ), }); diff --git a/packages/core/src/adapter/sqlite/sqlite.test.ts b/packages/core/src/adapter/sqlite/sqlite.test.ts index bd0f3e88..15738695 100644 --- a/packages/core/src/adapter/sqlite/sqlite.test.ts +++ b/packages/core/src/adapter/sqlite/sqlite.test.ts @@ -6,7 +6,7 @@ import { silentLogger } from "../../contracts/logger.ts"; import { type FieldsRecord, fieldBuilder as f } from "../../modules/view.ts"; import type { SqlDatabase } from "../sql/database.ts"; import type { SqlExecutor } from "../sql/sql-table.ts"; -import type { KeyedPerson } from "../storage/personal-key-store.ts"; +import { DEFAULT_PERSONAL_CATEGORY, type KeyedPerson } from "../storage/personal-key-store.ts"; import { testKeyedPerson as held, testPersonalKey as key, @@ -131,7 +131,7 @@ describe("the SQLite stores on node:sqlite", () => { idempotencyStoreContract({ create: storage }); snapshotStoreContract({ create: storage }); storageTransactionContract({ create: storage }); - personalKeyStoreContract({ create: async () => (await storage()).personalKeys }); + personalKeyStoreContract({ create: async () => storage() }); personStoreContract({ create: async () => (await storage()).persons }); inspectionContract({ create: async () => nodeSqlite().adapter }); subscriberFailureStoreContract({ create: async () => (await storage()).subscriberFailures }); @@ -180,8 +180,8 @@ describe("the person index of the SQLite stores", () => { it("keeps who each row names as its persons say, from its first write to its removal", async () => { const { adapter, db } = nodeSqlite(); const storage = await adapter.createStorage(); - await storage.personalKeys.add("c-1", key("1")); - await storage.personalKeys.add("c-2", key("2")); + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, key("1")); + await storage.personalKeys.add("c-2", DEFAULT_PERSONAL_CATEGORY, key("2")); await storage.snapshotStore.put({ aggregateType: "customer", aggregateId: "c-1", @@ -274,8 +274,10 @@ describe("the person index of the SQLite stores", () => { ]); }; + // A row's own `persons`, or the fingerprints a forget keeps, are read through `json_each`, a scan + // of the one value it is given, never of a store. const scans = (plan: readonly string[]): readonly string[] => - plan.filter((step) => step.startsWith("SCAN")); + plan.filter((step) => step.startsWith("SCAN") && !step.includes("VIRTUAL TABLE")); it("finds the rows that name a person by index, never reading every row of a store", async () => { const { adapter, db } = nodeSqlite(); diff --git a/packages/core/src/adapter/staged-event-store.test.ts b/packages/core/src/adapter/staged-event-store.test.ts index 9efa8da0..0ab26529 100644 --- a/packages/core/src/adapter/staged-event-store.test.ts +++ b/packages/core/src/adapter/staged-event-store.test.ts @@ -48,13 +48,13 @@ describe("createStagedEventStore", () => { it("reports the keys the base store's load opened, and none it was not given", async () => { const { base } = await setUp(); - const fingerprints = new Map([["c-1", "f-1"]]); + const opened = [{ person: "c-1", fingerprint: "f-1" }]; const opening = createStagedEventStore({ ...base, - load: async (args) => ({ ...(await base.load(args)), fingerprints }), + load: async (args) => ({ ...(await base.load(args)), opened }), }); - expect((await opening.load(order)).fingerprints).toBe(fingerprints); - expect(await createStagedEventStore(base).load(order)).not.toHaveProperty("fingerprints"); + expect((await opening.load(order)).opened).toBe(opened); + expect(await createStagedEventStore(base).load(order)).not.toHaveProperty("opened"); }); it("loads one stream: the base store's events plus what was staged for it, at the version both make", async () => { diff --git a/packages/core/src/adapter/staged-event-store.ts b/packages/core/src/adapter/staged-event-store.ts index fc8633a6..c4eaf336 100644 --- a/packages/core/src/adapter/staged-event-store.ts +++ b/packages/core/src/adapter/staged-event-store.ts @@ -101,7 +101,7 @@ export const createStagedEventStore: CreateStagedEventStoreFunction = ( ...staged.events.filter((event) => event.version >= fromVersion), ], version: staged.base + staged.events.length, - ...(stored.fingerprints === undefined ? {} : { fingerprints: stored.fingerprints }), + ...(stored.opened === undefined ? {} : { opened: stored.opened }), }; }, append: async ({ diff --git a/packages/core/src/adapter/storage/append-checks.test.ts b/packages/core/src/adapter/storage/append-checks.test.ts index c63d30a6..8ab75d57 100644 --- a/packages/core/src/adapter/storage/append-checks.test.ts +++ b/packages/core/src/adapter/storage/append-checks.test.ts @@ -1,7 +1,13 @@ import { describe, expect, it } from "vitest"; import { pendingEvent } from "../testing/fixtures.ts"; -import { appendPersonsOf, checkAppend, ForgottenSinceLoadError } from "./append-checks.ts"; +import { + appendPersonsOf, + checkAppend, + ForgottenSinceLoadError, + type KeyState, +} from "./append-checks.ts"; import type { AppendArgs } from "./event-store.ts"; +import type { KeyedPerson } from "./personal-key-store.ts"; const append = (names: Pick): AppendArgs => ({ aggregateType: "order", @@ -35,12 +41,12 @@ describe("appendPersonsOf", () => { describe("checkAppend", () => { it("checks nothing an append does not name", () => { expect(() => - checkAppend({ args: append({}), fingerprintOf: () => null, restricted: () => true }), + checkAppend({ args: append({}), keyOf: () => "gone", restricted: () => true }), ).not.toThrow(); expect(() => checkAppend({ args: append({ persons: [{ person: "c-1", fingerprint: null }] }), - fingerprintOf: () => null, + keyOf: () => "gone", restricted: () => true, }), ).not.toThrow(); @@ -51,7 +57,7 @@ describe("checkAppend", () => { try { checkAppend({ args: append({ persons: [{ person: "c-1", fingerprint: "f-1" }] }), - fingerprintOf: () => "f-2", + keyOf: () => "gone", restricted: () => false, }); } catch (error) { @@ -68,4 +74,22 @@ describe("checkAppend", () => { message: "Person c-1 was forgotten since order:1 was loaded at version 3", }); }); + + it("lets a key read in force through while it is in force, and one read retained while it is kept at all", () => { + const check = (read: KeyedPerson, state: KeyState) => () => + checkAppend({ + args: append({ persons: [read] }), + keyOf: (person, fingerprint) => + person === "c-1" && fingerprint === "f-1" ? state : "gone", + restricted: () => false, + }); + const inForce = { person: "c-1", fingerprint: "f-1" }; + const retained = { ...inForce, retained: true as const }; + expect(check(inForce, "held")).not.toThrow(); + expect(check(inForce, "retained")).toThrow(ForgottenSinceLoadError); + expect(check(inForce, "gone")).toThrow(ForgottenSinceLoadError); + expect(check(retained, "held")).not.toThrow(); + expect(check(retained, "retained")).not.toThrow(); + expect(check(retained, "gone")).toThrow(ForgottenSinceLoadError); + }); }); diff --git a/packages/core/src/adapter/storage/append-checks.ts b/packages/core/src/adapter/storage/append-checks.ts index 8a6143d9..975d6cae 100644 --- a/packages/core/src/adapter/storage/append-checks.ts +++ b/packages/core/src/adapter/storage/append-checks.ts @@ -44,13 +44,15 @@ export const appendPersonsOf: AppendPersonsOfFunction = ({ persons = [], touches touched: [...new Set(touches)], }); +/** + * A key of a person as the append's transaction reads it: `held` unless it is gone, `retained` + * when a forgetting kept it past it. + */ +export type KeyState = "held" | "retained" | "gone"; + export interface CheckAppendArgs { readonly args: AppendArgs; - /** - * The fingerprint of the key the person holds as the append's transaction reads it; `null` - * when they hold none. - */ - readonly fingerprintOf: (person: string) => string | null; + readonly keyOf: (person: string, fingerprint: string) => KeyState; /** * Whether the processing of the person's data is restricted, as the same transaction reads it. */ @@ -64,16 +66,18 @@ export interface CheckAppendFunction { /** * Throws `ProcessingRestrictedError` when a person `touches` names is restricted, and else * `ForgottenSinceLoadError` when a person `persons` names with a fingerprint no longer holds - * that key. A store calls it in the transaction that writes the events, on what that transaction + * that key, or holds it retained where it was read in force. A store calls it in the transaction that writes the events, on what that transaction * reads, so a `restrict` or `forget` committing beside the append is ordered before it or after * it, never across. */ -export const checkAppend: CheckAppendFunction = ({ args, fingerprintOf, restricted }) => { +export const checkAppend: CheckAppendFunction = ({ args, keyOf, restricted }) => { for (const person of args.touches ?? []) { if (restricted(person)) throw new ProcessingRestrictedError(person); } - for (const { person, fingerprint } of args.persons ?? []) { - if (fingerprint !== null && fingerprintOf(person) !== fingerprint) { + for (const { person, fingerprint, retained } of args.persons ?? []) { + if (fingerprint === null) continue; + const state = keyOf(person, fingerprint); + if (state === "gone" || (state === "retained" && retained !== true)) { throw new ForgottenSinceLoadError(args, person); } } diff --git a/packages/core/src/adapter/storage/dead-letter-store.ts b/packages/core/src/adapter/storage/dead-letter-store.ts index d7dcab8f..4a786b39 100644 --- a/packages/core/src/adapter/storage/dead-letter-store.ts +++ b/packages/core/src/adapter/storage/dead-letter-store.ts @@ -164,10 +164,12 @@ export interface DeadLetterStore { */ listByPerson(person: string): Promise; /** - * Removes every letter whose `persons` names `person`, whatever its status, and resolves to them - * in the order of `list`. A letter that names the person only in `person` stays. + * Removes every letter whose `persons` names `person` under a key outside `keep`, or read in + * the clear, whatever its status, and resolves to them in the order of `list`. Without `keep`, + * every letter whose `persons` names them. A letter that names the person only in `person` + * stays. */ - forget(person: string): Promise; + forget(person: string, keep?: readonly string[]): Promise; /** * The earliest `resolvedAt` of the letters held, `null` when none is resolved. */ diff --git a/packages/core/src/adapter/storage/event-store.ts b/packages/core/src/adapter/storage/event-store.ts index cc97a17d..4307cc34 100644 --- a/packages/core/src/adapter/storage/event-store.ts +++ b/packages/core/src/adapter/storage/event-store.ts @@ -19,8 +19,9 @@ export interface AppendArgs { * The people whose personal data the events depend on, each with the key it was read or is * sealed under: those the state the events were decided on held, and those whose fields the * events seal. The store appends only while each one named with a fingerprint still holds that - * key, and otherwise throws `ForgottenSinceLoadError`, a `ConcurrencyError`; one named with - * `null` needs nothing. Checked with `checkAppend`, in the transaction that writes the events. + * key, in force unless it was read `retained`, and otherwise throws `ForgottenSinceLoadError`, a + * `ConcurrencyError`; one named with `null` needs nothing. Checked with `checkAppend`, in the + * transaction that writes the events. */ readonly persons?: readonly KeyedPerson[]; /** @@ -45,10 +46,10 @@ export interface LoadResult { readonly events: readonly StoredEvent[]; readonly version: number; /** - * The fingerprint of the key that opened each person's sealed fields, set by the kernel, which - * opens them; a store leaves it out. + * Each person whose sealed fields opened, once per key that opened them, set by the kernel, + * which opens them; a store leaves it out. */ - readonly fingerprints?: ReadonlyMap; + readonly opened?: readonly KeyedPerson[]; } export interface ReadAllArgs { diff --git a/packages/core/src/adapter/storage/forget-ledger.ts b/packages/core/src/adapter/storage/forget-ledger.ts index 96039e84..21f6b9c8 100644 --- a/packages/core/src/adapter/storage/forget-ledger.ts +++ b/packages/core/src/adapter/storage/forget-ledger.ts @@ -2,12 +2,18 @@ import type { PersonalDataForgottenReason } from "../../modules/bounda-events.ts /** * One forgetting as the ledger keeps it: whom, which `PersonalDataForgotten` says so, when, why, - * and which keys it destroys. Ids, a time and fingerprints, nothing else: the ledger is a - * suppression list, the one thing about a forgotten person that has to outlive their data. The - * runtime writes it before the forgetting commits, so an entry may name an event no store holds. + * which of their categories it was asked for, and which keys it covers. Ids, names, a time and + * fingerprints, nothing else: the ledger is a suppression list, the one thing about a forgotten + * person that has to outlive their data. The runtime writes it before the forgetting commits, so + * an entry may name an event no store holds. */ export interface ForgetLedgerEntry { readonly personId: string; + /** + * The categories the forgetting was asked for, `null` for the person as a whole: forgetting + * them again forgets those again, as the request did. + */ + readonly categories: readonly string[] | null; /** * The id of the `PersonalDataForgotten` event, which no other copy of the store gives to another * event, unlike its position. @@ -19,9 +25,9 @@ export interface ForgetLedgerEntry { readonly forgottenAt: string; readonly reason: PersonalDataForgottenReason; /** - * The fingerprints of the keys the person held when the forgetting was asked: the reconciliation - * commits it again only while the person holds one of them, since any other key was given to - * them after. Absent from an entry written by a Bounda that did not record them, which any key + * The fingerprints of the keys in force the forgetting covered when it was asked: the + * reconciliation commits it again only while the person holds one of them, since any other key + * was given to them after. Absent from an entry written by a Bounda that did not record them, which any key * matches. */ readonly fingerprints?: readonly string[]; diff --git a/packages/core/src/adapter/storage/idempotency-store.ts b/packages/core/src/adapter/storage/idempotency-store.ts index b07e07b3..bcd6f41b 100644 --- a/packages/core/src/adapter/storage/idempotency-store.ts +++ b/packages/core/src/adapter/storage/idempotency-store.ts @@ -46,10 +46,11 @@ export interface PurgeIdempotencyRecordsArgs { * its outcome, and the other rejects with `IdempotencyKeyTakenError`, which rolls back whatever * the recording was part of. * - * A record that names people lives only as long as their keys, as a snapshot does: `put` records - * nothing unless every person it names with a fingerprint holds the key of that fingerprint (one - * named with `null` was read in the clear and needs none), checked in the same statement, and - * `forget` removes every record that names a person. + * A record that names people lives only as long as their keys stay in force, as a snapshot does: + * `put` records nothing unless every person it names with a fingerprint holds the key of that + * fingerprint in force (one named with `null` was read in the clear and needs none), checked in + * the same statement, and `forget` removes every record that names the person under a key outside + * `keep`, or read in the clear. */ export interface IdempotencyStore { /** @@ -65,5 +66,5 @@ export interface IdempotencyStore { * Removes the records expired by `now`, and resolves to how many. */ purge(args: PurgeIdempotencyRecordsArgs): Promise; - forget(person: string): Promise; + forget(person: string, keep?: readonly string[]): Promise; } diff --git a/packages/core/src/adapter/storage/person-store.ts b/packages/core/src/adapter/storage/person-store.ts index 92d9ea48..204b515a 100644 --- a/packages/core/src/adapter/storage/person-store.ts +++ b/packages/core/src/adapter/storage/person-store.ts @@ -18,6 +18,11 @@ export interface StoredPerson { * when none has been. */ readonly lastActiveAt: string | null; + /** + * The position of the last event recorded on the row, `0` before any: what names them past it + * is not recorded yet. + */ + readonly position: number; /** * The timestamp of the `PersonalDataForgotten` recorded after their last activity; `null` when * an event of theirs came after it, or none was recorded. diff --git a/packages/core/src/adapter/storage/personal-key-store.ts b/packages/core/src/adapter/storage/personal-key-store.ts index 36f71bde..8123965b 100644 --- a/packages/core/src/adapter/storage/personal-key-store.ts +++ b/packages/core/src/adapter/storage/personal-key-store.ts @@ -1,5 +1,11 @@ import type { Hold } from "./hold.ts"; +/** + * The category of a field marked `personal` without one: what every personal field was before + * categories existed, and what a store fills in for a key or an envelope that names none. + */ +export const DEFAULT_PERSONAL_CATEGORY: "default" = "default"; + /** * A person's key as the key store keeps it: the key, as opaque text the kernel chooses; its * fingerprint, which names the key without revealing it and stays the same however the key is @@ -12,15 +18,39 @@ export interface StoredPersonalKey { } /** - * A person, the key the store keeps for them, and the wait the rotation set after it could not - * rewrap that key, `null` when none is set. + * One of a person's keys, as `list` hands it out: the category it seals, the key, when it was + * kept past a forgetting under a legal obligation (`null` while it is in force), and the last + * activity of the person recorded on it (`null` before any), which a key kept past a forgetting + * stops following. + */ +export interface HeldPersonalKey { + readonly category: string; + readonly key: StoredPersonalKey; + readonly retainedSince: string | null; + readonly lastActiveAt: string | null; +} + +/** + * A person, one of their keys and the category it seals, and the wait the rotation set after it + * could not rewrap that key, `null` when none is set. */ export interface PersonalKeyRow { readonly person: string; + readonly category: string; readonly key: StoredPersonalKey; readonly hold: Hold | null; } +/** + * A key a category's retention may destroy: the person, the key, and the wait the retention set + * after it could not destroy it, `null` when none is set. + */ +export interface DueKey extends Omit { + readonly fingerprint: string; + readonly lastActiveAt: string; + readonly retainedSince: string | null; +} + /** * A key wrapped anew: its text and the id of the key it is now wrapped under. */ @@ -45,24 +75,80 @@ export interface NextKeyHoldEndArgs extends Pick { readonly after: Date; } +export interface RetainKeyArgs { + /** + * When the forgetting kept it, as an ISO timestamp. + */ + readonly since: string; + /** + * The person's last activity as the forgetting found it, which the key keeps from then on. + */ + readonly lastActiveAt: string; +} + +export interface ListDueArgs { + readonly category: string; + /** + * Only the keys last active at or before it; every one when omitted. + */ + readonly activeBefore?: Date; + /** + * Leaves out the keys held past it; holds are not looked at when omitted. + */ + readonly releasedBy?: Date; + readonly limit: number; +} + +export interface ListRetainedArgs { + /** + * Leaves out the keys of these categories: those a legal obligation still keeps. + */ + readonly except: readonly string[]; + /** + * Leaves out the keys held past it; holds are not looked at when omitted. + */ + readonly releasedBy?: Date; + readonly limit: number; +} + +export interface NextRetainedHoldEndArgs { + readonly except: readonly string[]; + /** + * Only the holds that end after it. + */ + readonly after: Date; +} + +export interface NextDueHoldEndArgs { + readonly category: string; + /** + * Only the holds that end after it. + */ + readonly after: Date; +} + /** * A person whose personal data a state holds in the clear, with the fingerprint of the key it was * read under; `null` when it was read from events not sealed yet, staged by the same unit of work. - * A person forgotten and given a new key since holds another fingerprint, so what the state holds + * `retained` when that key was kept past a forgetting when it was read: such a read holds while + * the key is kept, where one made while the key was in force holds only while it still is. A + * person forgotten and given a new key since holds another fingerprint, so what the state holds * of them must not be written back. */ export interface KeyedPerson { readonly person: string; readonly fingerprint: string | null; + readonly retained?: true; } /** - * A personal value as it is stored: the ciphertext, its nonce, the person whose key opens it and - * the fingerprint of that key, so a value sealed under a key the person no longer has is told - * from one their current key opens. + * A personal value as it is stored: the ciphertext, its nonce, the person whose key opens it, the + * category of that key, absent for the default one, and the fingerprint of the key, so a value + * sealed under a key the person no longer has is told from one their current key opens. */ export interface PersonalEnvelope { readonly person: string; + readonly category?: string; readonly key: string; readonly iv: string; readonly data: string; @@ -86,33 +172,72 @@ export const keyedPersonId: KeyedPersonIdFunction = ({ person, fingerprint }) => JSON.stringify([person, fingerprint]); export interface NamesPersonFunction { - (row: { readonly persons?: readonly KeyedPerson[] }, person: string): boolean; + ( + row: { readonly persons?: readonly KeyedPerson[] }, + person: string, + keep?: readonly string[], + ): boolean; } -// Under any key: what `forget` removes for a store that filters its rows itself. -export const namesPerson: NamesPersonFunction = (row, person) => - row.persons?.some((held) => held.person === person) ?? false; +/** + * Whether the row names the person under a key outside `keep`, or read in the clear: what + * `forget` removes, for a store that filters its rows itself. + */ +export const namesPerson: NamesPersonFunction = (row, person, keep = []) => + row.persons?.some( + (held) => + held.person === person && (held.fingerprint === null || !keep.includes(held.fingerprint)), + ) ?? false; /** - * The keys the personal fields of events are encrypted with, one per person. `get` is `null` for - * a person without one, as for one forgotten. `add` stores `key` for `person` unless one is in - * force already and resolves with the one in force, so two writers that find none keep the same - * key. `remove` forgets the key and reports whether there was one; what was encrypted with it can - * never be read again. `listWrappedUnder` lists, by person, keys wrapped under one of `kids`, and - * `rewrap` replaces the key `from` of a person with the same key wrapped as `to` says, keeping - * its fingerprint and clearing its hold, only while the person still holds `from`, and reports - * whether it did: a key forgotten or replaced meanwhile is never written back. `hold` sets the - * hold of the key `from` on the same condition, and `nextHoldEnd` answers the earliest end of a - * hold past `after` among the keys wrapped under one of `kids`, `null` when none is held past it. + * The keys personal fields are encrypted with: one in force per person and category, and those + * a forgetting kept past it under a legal obligation, `retained`. A key is named by the person + * and its fingerprint. + * + * `get` is the key in force for the person and category, `null` when there is none. `find` is the + * key of that fingerprint, in force or retained, `null` once it is gone. `list` is every key the + * person holds. `add` stores `key` as the one in force for the person and category unless one is + * in force already and resolves with the one in force, so two writers that find none keep the + * same key. `remove` forgets a key and reports whether there was one; what was encrypted with it + * can never be read again. `retain` keeps a key in force past a forgetting, and reports whether it + * did: from then on it is no longer in force, and a later `add` for its category stores another. + * `touch` records the person's last activity on every key of theirs in force, never moving one + * back. + * + * `listDue` lists, least recently active first, the keys of a category, in force or retained, + * whose person is not restricted, and `holdDue` sets the hold of one of them, which only the + * retention reads, while it still records `lastActiveAt`. `nextDueHoldEnd` answers the earliest + * end of such a hold past `after`, `null` when none is held past it. + * + * `listRetained` lists, kept longest first, the keys a forgetting retained whose category is not + * one of `except`, whose person is not restricted: what no legal obligation keeps any more, as a + * category whose basis changed. Their holds are those of `holdDue`, and `nextRetainedHoldEnd` + * answers the earliest end of one past `after`. * - * Stores that keep personal data in the clear, as snapshots do, compare a person's fingerprint - * with the one in force in the same statement as they write, so the store must keep it where - * they can read it. + * `listWrappedUnder` lists, by person, keys wrapped under one of `kids`, and `rewrap` replaces + * the key `from` of a person with the same key wrapped as `to` says, keeping its fingerprint and + * clearing its hold, only while the person still holds `from`, and reports whether it did: a key + * forgotten or replaced meanwhile is never written back. `hold` sets the hold of the key `from` on + * the same condition, and `nextHoldEnd` answers the earliest end of a hold past `after` among the + * keys wrapped under one of `kids`, `null` when none is held past it. + * + * Stores that keep personal data in the clear, as snapshots do, compare a person's fingerprints + * with the keys in force in the same statement as they write, so the store must keep them where + * they can read them. */ export interface PersonalKeyStore { - get(person: string): Promise; - add(person: string, key: StoredPersonalKey): Promise; - remove(person: string): Promise; + get(person: string, category: string): Promise; + find(person: string, fingerprint: string): Promise; + list(person: string): Promise; + add(person: string, category: string, key: StoredPersonalKey): Promise; + remove(person: string, fingerprint: string): Promise; + retain(person: string, fingerprint: string, args: RetainKeyArgs): Promise; + touch(person: string, lastActiveAt: string): Promise; + listDue(args: ListDueArgs): Promise; + holdDue(person: string, fingerprint: string, lastActiveAt: string, hold: Hold): Promise; + nextDueHoldEnd(args: NextDueHoldEndArgs): Promise; + listRetained(args: ListRetainedArgs): Promise; + nextRetainedHoldEnd(args: NextRetainedHoldEndArgs): Promise; listWrappedUnder(args: ListWrappedUnderArgs): Promise; rewrap(person: string, from: StoredPersonalKey, to: RewrappedKey): Promise; hold(person: string, from: StoredPersonalKey, hold: Hold): Promise; @@ -126,4 +251,17 @@ export interface PersonalKeyStore { * database went back to a copy, whose holds count failures the copy undid. */ releaseHolds(): Promise; + /** + * How many keys `listDue` lists with `activeBefore`, whatever their holds. + */ + countDue(args: Pick): Promise; + /** + * How many keys `listRetained` lists, whatever their holds. + */ + countRetained(args: Pick): Promise; + /** + * Clears the hold of every key `holdDue` held, and resolves to how many it cleared, as + * `releaseHolds` does for the rotation's. + */ + releaseDueHolds(): Promise; } diff --git a/packages/core/src/adapter/storage/scheduler.ts b/packages/core/src/adapter/storage/scheduler.ts index b80fc7c0..0f367ed6 100644 --- a/packages/core/src/adapter/storage/scheduler.ts +++ b/packages/core/src/adapter/storage/scheduler.ts @@ -172,9 +172,10 @@ export interface Scheduler { */ listByPerson(person: string): Promise; /** - * Removes every command whose `persons` names `person`, claimed or not, and resolves to them in - * the order of `list`: a worker running one then finds its claim lost, so what it writes, a dead - * letter included, is rolled back. + * Removes every command whose `persons` names `person` under a key outside `keep`, or read in + * the clear, claimed or not, and resolves to them in the order of `list`: a worker running one + * then finds its claim lost, so what it writes, a dead letter included, is rolled back. Without + * `keep`, every command that names them. */ - forget(person: string): Promise; + forget(person: string, keep?: readonly string[]): Promise; } diff --git a/packages/core/src/adapter/storage/snapshot-store.ts b/packages/core/src/adapter/storage/snapshot-store.ts index 6e328c98..725ec69f 100644 --- a/packages/core/src/adapter/storage/snapshot-store.ts +++ b/packages/core/src/adapter/storage/snapshot-store.ts @@ -19,17 +19,19 @@ export interface Snapshot extends StreamIdentity { * so a kernel that finds one missing, stale or doubtful folds the stream from its first event * instead. `get` is `null` for a stream without one. `remove` forgets a stream's snapshot. * - * A snapshot holds personal data in the clear, so it lives only as long as its persons' keys: - * `put` writes nothing unless every person it names with a fingerprint holds, in the storage's - * `personalKeys`, the key of that fingerprint, checked in the same statement (one named with - * `null` was read in the clear and needs none); and `forget` removes every snapshot that names a - * person. A transaction that removes a key and then forgets its snapshots leaves none behind, - * whatever `put` runs beside it, and a key given to the person afterwards is never the one a - * snapshot was read under. A `put` older than the snapshot of the same fingerprint is ignored. + * A snapshot holds personal data in the clear, so it lives only as long as its persons' keys stay + * in force: `put` writes nothing unless every person it names with a fingerprint holds, in the + * storage's `personalKeys`, the key of that fingerprint in force, neither gone nor retained, + * checked in the same statement (one named with `null` was read in the clear and needs none); and + * `forget` removes every snapshot that names the person under a key outside `keep`, or read in the + * clear, every one that names them without `keep`. A transaction that removes or retains a key and + * then forgets the snapshots outside the keys still in force leaves none behind, whatever `put` + * runs beside it, and a key given to the person afterwards is never the one a snapshot was read + * under. A `put` older than the snapshot of the same fingerprint is ignored. */ export interface SnapshotStore { get(stream: StreamIdentity): Promise; put(snapshot: Snapshot): Promise; remove(stream: StreamIdentity): Promise; - forget(person: string): Promise; + forget(person: string, keep?: readonly string[]): Promise; } diff --git a/packages/core/src/adapter/testing/dead-letter-store.contract.ts b/packages/core/src/adapter/testing/dead-letter-store.contract.ts index 5b00c7d4..9e03d69b 100644 --- a/packages/core/src/adapter/testing/dead-letter-store.contract.ts +++ b/packages/core/src/adapter/testing/dead-letter-store.contract.ts @@ -1,6 +1,7 @@ import { beforeEach, describe, expect, it } from "vitest"; import { DeadLetterSettledError } from "../../contracts/errors.ts"; import type { DeadLetterStore, NewDeadLetter } from "../storage/dead-letter-store.ts"; +import type { KeyedPerson } from "../storage/personal-key-store.ts"; import { testKeyedPerson as held, testSealedValue as sealed } from "./fixtures.ts"; // When a test resolves a letter. @@ -302,6 +303,23 @@ export const deadLetterStoreContract: DeadLetterStoreContractFunction = ({ creat expect(await store.listByPerson("c-1")).toEqual([]); }); + it("forgets the letters that name a person under a key outside those kept, or in the clear", async () => { + const scheduled = (id: string, persons: readonly KeyedPerson[]) => + letter(id, { + kind: "scheduled", + handler: "PlaceOrder", + eventType: "PlaceOrder", + payload: { orderId: id }, + persons, + }); + await store.add(scheduled("a", [held("c-1", "1")])); + await store.add(scheduled("b", [held("c-1", "2")])); + await store.add(scheduled("c", [held("c-1", null)])); + const forgotten = await store.forget("c-1", ["f-2"]); + expect(forgotten.map((kept) => kept.id).sort()).toEqual(["a", "c"]); + expect((await store.list()).map((kept) => kept.id)).toEqual(["b"]); + }); + it("leaves the letters that name a person only as the person of their work", async () => { await store.add( letter("policy", { diff --git a/packages/core/src/adapter/testing/earlier-rows.ts b/packages/core/src/adapter/testing/earlier-rows.ts index ef7d393b..329b1059 100644 --- a/packages/core/src/adapter/testing/earlier-rows.ts +++ b/packages/core/src/adapter/testing/earlier-rows.ts @@ -1,5 +1,6 @@ import { expect } from "vitest"; import type { Storage } from "../adapter.ts"; +import { DEFAULT_PERSONAL_CATEGORY as D } from "../storage/personal-key-store.ts"; import type { EarlierTables, EarlierVersion } from "./earlier-schemas.ts"; import { testKeyedPerson as held, @@ -152,6 +153,9 @@ export const expectEarlierRowsTakenIn: ExpectEarlierRowsTakenInFunction = async reason: "requested", cancelledCommands: [], removedDeadLetters: [], + categories: ["default"], + retained: [], + whole: true, }, person: "p-1", }, @@ -210,7 +214,13 @@ export const expectEarlierRowsTakenIn: ExpectEarlierRowsTakenInFunction = async expect((await storage.scheduler.listByPerson("p-2")).map(({ dedupeKey }) => dedupeKey)).toEqual([ "s-1", ]); - expect(await storage.personalKeys.get("p-2")).toEqual(testPersonalKey("1")); + expect(await storage.personalKeys.get("p-2", D)).toEqual(testPersonalKey("1")); + expect(await storage.personalKeys.find("p-2", testPersonalKey("1").fingerprint)).toEqual({ + category: D, + key: testPersonalKey("1"), + retainedSince: null, + lastActiveAt: null, + }); expect( await storage.snapshotStore.get({ aggregateType: "order", aggregateId: "1" }), ).not.toBeNull(); diff --git a/packages/core/src/adapter/testing/event-store.contract.ts b/packages/core/src/adapter/testing/event-store.contract.ts index a5a09904..ea01bd2d 100644 --- a/packages/core/src/adapter/testing/event-store.contract.ts +++ b/packages/core/src/adapter/testing/event-store.contract.ts @@ -3,6 +3,7 @@ import { ConcurrencyError, ProcessingRestrictedError } from "../../contracts/err import type { Storage } from "../adapter.ts"; import { ForgottenSinceLoadError } from "../storage/append-checks.ts"; import type { AppendArgs, EventStore } from "../storage/event-store.ts"; +import { DEFAULT_PERSONAL_CATEGORY as D } from "../storage/personal-key-store.ts"; import { pendingEvent, testPersonalKey } from "./fixtures.ts"; export interface EventStoreContractArgs { @@ -60,10 +61,10 @@ export const eventStoreContract: EventStoreContractFunction = ({ create, concurr }; it("appends only while each person it names with a key still holds that key", async () => { - await storage.personalKeys.add("c-1", testPersonalKey("one")); + await storage.personalKeys.add("c-1", D, testPersonalKey("one")); const first = { person: "c-1", fingerprint: testPersonalKey("one").fingerprint }; await naming(1, { persons: [first, { person: "c-2", fingerprint: null }] }); - await storage.personalKeys.remove("c-1"); + await storage.personalKeys.remove("c-1", "f-one"); const refused = await naming(2, { persons: [first] }).catch((error: unknown) => error); expect(refused).toBeInstanceOf(ForgottenSinceLoadError); expect(refused).toBeInstanceOf(ConcurrencyError); @@ -74,7 +75,7 @@ export const eventStoreContract: EventStoreContractFunction = ({ create, concurr expectedVersion: 1, actualVersion: 1, }); - await storage.personalKeys.add("c-1", testPersonalKey("two")); + await storage.personalKeys.add("c-1", D, testPersonalKey("two")); await expect(naming(2, { persons: [first] })).rejects.toBeInstanceOf(ForgottenSinceLoadError); await expect( storage.transact((tx) => naming(2, { persons: [first] }, tx.eventStore)), @@ -86,6 +87,23 @@ export const eventStoreContract: EventStoreContractFunction = ({ create, concurr expect((await store.load({ aggregateType: "order", aggregateId: "1" })).version).toBe(2); }); + it("appends on a key kept past a forgetting only for what was read from it once kept", async () => { + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("one")); + const read = { person: "c-1", fingerprint: testPersonalKey("one").fingerprint }; + await naming(1, { persons: [read] }); + await storage.personalKeys.retain("c-1", "f-one", { + since: "2026-02-01T00:00:00.000Z", + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + await expect(naming(2, { persons: [read] })).rejects.toBeInstanceOf(ForgottenSinceLoadError); + await naming(2, { persons: [{ ...read, retained: true }] }); + await storage.personalKeys.remove("c-1", "f-one"); + await expect(naming(3, { persons: [{ ...read, retained: true }] })).rejects.toBeInstanceOf( + ForgottenSinceLoadError, + ); + expect((await store.load({ aggregateType: "order", aggregateId: "1" })).version).toBe(2); + }); + it("refuses to append while a person it touches is restricted, before any key it names", async () => { await storage.persons.restrict("c-1", now); const refused = await naming(1, { @@ -129,14 +147,14 @@ export const eventStoreContract: EventStoreContractFunction = ({ create, concurr it.skipIf(!concurrent)( "refuses an append that a transaction forgetting its person held up", async () => { - await storage.personalKeys.add("c-1", testPersonalKey("one")); + await storage.personalKeys.add("c-1", D, testPersonalKey("one")); const { held, release } = holding(); let removed = (): void => {}; const removing = new Promise((resolve) => { removed = resolve; }); const forget = storage.transact(async (tx) => { - await tx.personalKeys.remove("c-1"); + await tx.personalKeys.remove("c-1", "f-one"); removed(); await held; }); diff --git a/packages/core/src/adapter/testing/forget-ledger.contract.ts b/packages/core/src/adapter/testing/forget-ledger.contract.ts index 6936d79e..1907be0b 100644 --- a/packages/core/src/adapter/testing/forget-ledger.contract.ts +++ b/packages/core/src/adapter/testing/forget-ledger.contract.ts @@ -12,10 +12,16 @@ export interface ForgetLedgerContractFunction { (args: ForgetLedgerContractArgs): void; } -const entry = (personId: string, eventId: string, minute: number): ForgetLedgerEntry => ({ +const entry = ( + personId: string, + eventId: string, + minute: number, + categories: readonly string[] | null = null, +): ForgetLedgerEntry => ({ personId, eventId, forgottenAt: new Date(Date.UTC(2026, 10, 7, 12, minute)).toISOString(), + categories, reason: "requested", fingerprints: [`k-${personId}`], }); @@ -44,6 +50,17 @@ export const forgetLedgerContract: ForgetLedgerContractFunction = ({ create }) = ledger = await ledgerFor(); }); + it("keeps the categories each forgetting asked for, none for a person forgotten as a whole", async () => { + const entries = [ + entry("p-1", "e-1", 1), + entry("p-2", "e-2", 2, ["marketing"]), + entry("p-3", "e-3", 3, ["marketing", "invoices"]), + entry("p-4", "e-4", 4, []), + ]; + await ledger.append(entries); + expect(await readAll(ledger, 10)).toEqual(entries); + }); + it("reads nothing from an empty ledger", async () => { expect(await ledger.read({ limit: 10 })).toEqual([]); expect( diff --git a/packages/core/src/adapter/testing/idempotency-store.contract.ts b/packages/core/src/adapter/testing/idempotency-store.contract.ts index 741259e7..c25936d0 100644 --- a/packages/core/src/adapter/testing/idempotency-store.contract.ts +++ b/packages/core/src/adapter/testing/idempotency-store.contract.ts @@ -3,6 +3,7 @@ import { IdempotencyKeyTakenError } from "../../contracts/errors.ts"; import type { Storage } from "../adapter.ts"; import type { IdempotencyStore } from "../storage/idempotency-store.ts"; import type { KeyedPerson, PersonalKeyStore } from "../storage/personal-key-store.ts"; +import { DEFAULT_PERSONAL_CATEGORY as D } from "../storage/personal-key-store.ts"; import { testKeyedPerson as held, testPersonalKey as stored, testOutcome } from "./fixtures.ts"; export interface IdempotencyStoreContractArgs { @@ -48,14 +49,14 @@ export const idempotencyStoreContract: IdempotencyStoreContractFunction = ({ cre it("records a record that names people only while each of them has a key", async () => { const persons = [held("c-1", "1"), held("c-2", "2"), held("c-3", null)]; const named = { ...record("k1"), outcome: rejection, persons, now }; - await keys.add("c-1", stored("1")); + await keys.add("c-1", D, stored("1")); await store.put(named); expect(await store.get({ key: "k1", now })).toBeNull(); - await keys.add("c-2", stored("2")); + await keys.add("c-2", D, stored("2")); await store.put(named); const { now: _now, ...kept } = named; expect(await store.get({ key: "k1", now })).toEqual(kept); - await keys.remove("c-1"); + await keys.remove("c-1", "f-1"); await expect(store.put({ ...named, fingerprint: "other" })).rejects.toBeInstanceOf( IdempotencyKeyTakenError, ); @@ -68,7 +69,7 @@ export const idempotencyStoreContract: IdempotencyStoreContractFunction = ({ cre persons, now, }); - await keys.add("c-1", stored("new")); + await keys.add("c-1", D, stored("new")); await store.put(named(held("c-1", "old"))); expect(await store.get({ key: "k1", now })).toBeNull(); await store.put(named(held("c-1", "old"), held("c-1", "new"))); @@ -81,8 +82,8 @@ export const idempotencyStoreContract: IdempotencyStoreContractFunction = ({ cre }); it("forgets every record that names a person, and no other", async () => { - await keys.add("c-1", stored("1")); - await keys.add("c-2", stored("2")); + await keys.add("c-1", D, stored("1")); + await keys.add("c-2", D, stored("2")); const c1 = held("c-1", "1"); const c2 = held("c-2", "2"); await store.put({ ...record("k1"), outcome: rejection, persons: [c1], now }); @@ -97,6 +98,22 @@ export const idempotencyStoreContract: IdempotencyStoreContractFunction = ({ cre expect(left).toEqual([null, null, "k3", "k4"]); }); + it("forgets the records that name a person under a key outside those kept, and records none under a key kept past a forgetting", async () => { + await keys.add("c-1", D, stored("1")); + await keys.add("c-1", "invoices", stored("2")); + await store.put({ ...record("k1"), outcome: rejection, persons: [held("c-1", "1")], now }); + await store.put({ ...record("k2"), outcome: rejection, persons: [held("c-1", "2")], now }); + await store.forget("c-1", ["f-2"]); + expect(await store.get({ key: "k1", now })).toBeNull(); + expect((await store.get({ key: "k2", now }))?.key).toBe("k2"); + await keys.retain("c-1", "f-2", { + since: "2026-02-01T00:00:00.000Z", + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + await store.put({ ...record("k3"), outcome: rejection, persons: [held("c-1", "2")], now }); + expect(await store.get({ key: "k3", now })).toBeNull(); + }); + it("answers a record as it was put, and null for a key it never saw", async () => { await store.put({ ...record("k1"), now }); expect(await store.get({ key: "k1", now })).toEqual(record("k1")); diff --git a/packages/core/src/adapter/testing/inspection.contract.ts b/packages/core/src/adapter/testing/inspection.contract.ts index bb64691a..cda913e7 100644 --- a/packages/core/src/adapter/testing/inspection.contract.ts +++ b/packages/core/src/adapter/testing/inspection.contract.ts @@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest"; import { silentLogger } from "../../contracts/logger.ts"; import type { Adapter } from "../adapter.ts"; import type { InspectedTable } from "../storage/inspection.ts"; +import { DEFAULT_PERSONAL_CATEGORY } from "../storage/personal-key-store.ts"; import { pendingEvent, testCommand, testContext, testDeadLetter, testOutcome } from "./fixtures.ts"; import { type ContractRow, contractFields } from "./table.contract.ts"; @@ -43,8 +44,16 @@ export const inspectionContract: InspectionContractFunction = ({ create }) => { expectedVersion: 0, events: [pendingEvent({ aggregateId: "o-1", version: 1, payload: { note: "ring twice" } })], }); - await storage.personalKeys.add("c-2", { key: "a2V5Mg==", fingerprint: "ZnAy", kid: null }); - await storage.personalKeys.add("c-1", { key: "a2V5", fingerprint: "ZnA=", kid: null }); + await storage.personalKeys.add("c-2", DEFAULT_PERSONAL_CATEGORY, { + key: "a2V5Mg==", + fingerprint: "ZnAy", + kid: null, + }); + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, { + key: "a2V5", + fingerprint: "ZnA=", + kid: null, + }); await storage.checkpointStore.set("orders", 1); await storage.inboxLedger.tryClaim({ handler: "order.p", eventId: "e1", now, leaseMs: 1 }); await storage.scheduler.schedule({ @@ -161,6 +170,7 @@ export const inspectionContract: InspectionContractFunction = ({ create }) => { personId: "c-1", eventId: "e-1", forgottenAt: "2026-11-07T12:00:00.000Z", + categories: null, reason: "requested", fingerprints: [], }, diff --git a/packages/core/src/adapter/testing/person-store.contract.ts b/packages/core/src/adapter/testing/person-store.contract.ts index 39b3d609..30fb967a 100644 --- a/packages/core/src/adapter/testing/person-store.contract.ts +++ b/packages/core/src/adapter/testing/person-store.contract.ts @@ -38,6 +38,7 @@ export const personStoreContract: PersonStoreContractFunction = ({ create }) => person: "c-1", restrictedSince: "2026-01-01T00:00:00.000Z", lastActiveAt: null, + position: 0, forgottenAt: null, hold: null, }); @@ -62,6 +63,7 @@ export const personStoreContract: PersonStoreContractFunction = ({ create }) => person: "c-1", restrictedSince: null, lastActiveAt: at(1), + position: 3, forgottenAt: null, hold: null, }); @@ -74,6 +76,7 @@ export const personStoreContract: PersonStoreContractFunction = ({ create }) => person: "c-2", restrictedSince: null, lastActiveAt: null, + position: 9, forgottenAt: at(4), hold: null, }); @@ -127,6 +130,7 @@ export const personStoreContract: PersonStoreContractFunction = ({ create }) => person: "c-1", restrictedSince: null, lastActiveAt: at(1), + position: 3, forgottenAt: null, hold: null, }, diff --git a/packages/core/src/adapter/testing/personal-key-store.contract.ts b/packages/core/src/adapter/testing/personal-key-store.contract.ts index b2467d49..050edd10 100644 --- a/packages/core/src/adapter/testing/personal-key-store.contract.ts +++ b/packages/core/src/adapter/testing/personal-key-store.contract.ts @@ -1,9 +1,16 @@ import { beforeEach, describe, expect, it } from "vitest"; -import type { PersonalKeyStore } from "../storage/personal-key-store.ts"; +import type { Storage } from "../adapter.ts"; +import { + DEFAULT_PERSONAL_CATEGORY as D, + type PersonalKeyStore, +} from "../storage/personal-key-store.ts"; import { testPersonalKey as stored, testWrappedPersonalKey as wrapped } from "./fixtures.ts"; export interface PersonalKeyStoreContractArgs { - readonly create: () => Promise; + /** + * A fresh, empty storage: the keys, and the persons whose restriction `listDue` reads. + */ + readonly create: () => Promise>; } export interface PersonalKeyStoreContractFunction { @@ -17,97 +24,100 @@ export interface PersonalKeyStoreContractFunction { export const personalKeyStoreContract: PersonalKeyStoreContractFunction = ({ create }) => { describe("personal key store contract", () => { let store: PersonalKeyStore; + let persons: Storage["persons"]; beforeEach(async () => { - store = await create(); + ({ personalKeys: store, persons } = await create()); }); it("has no key for a person it never met", async () => { - expect(await store.get("c-1")).toBeNull(); - expect(await store.remove("c-1")).toBe(false); + expect(await store.get("c-1", D)).toBeNull(); + expect(await store.find("c-1", "f-first")).toBeNull(); + expect(await store.list("c-1")).toEqual([]); + expect(await store.remove("c-1", "f-first")).toBe(false); }); it("keeps the first key added for a person, with its fingerprint, and answers every add with it", async () => { - expect(await store.add("c-1", stored("first"))).toEqual(stored("first")); - expect(await store.add("c-1", stored("second"))).toEqual(stored("first")); - expect(await store.get("c-1")).toEqual(stored("first")); - expect(await store.add("c-2", stored("other"))).toEqual(stored("other")); - expect(await store.get("c-2")).toEqual(stored("other")); + expect(await store.add("c-1", D, stored("first"))).toEqual(stored("first")); + expect(await store.add("c-1", D, stored("second"))).toEqual(stored("first")); + expect(await store.get("c-1", D)).toEqual(stored("first")); + expect(await store.add("c-2", D, stored("other"))).toEqual(stored("other")); + expect(await store.get("c-2", D)).toEqual(stored("other")); }); it("keeps one key when several writers add one at once", async () => { const added = await Promise.all( - ["a", "b", "c"].map((name) => store.add("c-1", stored(name))), + ["a", "b", "c"].map((name) => store.add("c-1", D, stored(name))), ); - const kept = await store.get("c-1"); + const kept = await store.get("c-1", D); expect(kept).not.toBeNull(); expect(added).toEqual([kept, kept, kept]); }); it("forgets a key for good, and takes a new one for the same person afterwards", async () => { - await store.add("c-1", stored("first")); - expect(await store.remove("c-1")).toBe(true); - expect(await store.get("c-1")).toBeNull(); - expect(await store.remove("c-1")).toBe(false); - expect(await store.add("c-1", stored("again"))).toEqual(stored("again")); - expect(await store.get("c-1")).toEqual(stored("again")); + await store.add("c-1", D, stored("first")); + expect(await store.remove("c-1", "f-first")).toBe(true); + expect(await store.get("c-1", D)).toBeNull(); + expect(await store.remove("c-1", "f-first")).toBe(false); + expect(await store.add("c-1", D, stored("again"))).toEqual(stored("again")); + expect(await store.get("c-1", D)).toEqual(stored("again")); }); it("keeps the id of the key a key is wrapped under", async () => { - await store.add("c-1", wrapped("first", "kid-1")); - expect(await store.get("c-1")).toEqual(wrapped("first", "kid-1")); + await store.add("c-1", D, wrapped("first", "kid-1")); + expect(await store.get("c-1", D)).toEqual(wrapped("first", "kid-1")); }); it("lists the keys wrapped under the ids asked for, unwrapped ones with null, by person and up to the limit", async () => { - await store.add("c-3", wrapped("3", "kid-1")); - await store.add("c-1", wrapped("1", "kid-1")); - await store.add("c-2", wrapped("2", "kid-2")); - await store.add("c-4", stored("4")); - await store.add("c-5", wrapped("5", "kid-3")); + await store.add("c-3", D, wrapped("3", "kid-1")); + await store.add("c-1", D, wrapped("1", "kid-1")); + await store.add("c-2", D, wrapped("2", "kid-2")); + await store.add("c-4", D, stored("4")); + await store.add("c-5", D, wrapped("5", "kid-3")); expect(await store.listWrappedUnder({ kids: ["kid-1"], limit: 10 })).toEqual([ - { person: "c-1", key: wrapped("1", "kid-1"), hold: null }, - { person: "c-3", key: wrapped("3", "kid-1"), hold: null }, + { person: "c-1", category: D, key: wrapped("1", "kid-1"), hold: null }, + { person: "c-3", category: D, key: wrapped("3", "kid-1"), hold: null }, ]); expect(await store.listWrappedUnder({ kids: ["kid-2", null], limit: 10 })).toEqual([ - { person: "c-2", key: wrapped("2", "kid-2"), hold: null }, - { person: "c-4", key: stored("4"), hold: null }, + { person: "c-2", category: D, key: wrapped("2", "kid-2"), hold: null }, + { person: "c-4", category: D, key: stored("4"), hold: null }, ]); expect(await store.listWrappedUnder({ kids: ["kid-1", "kid-2"], limit: 2 })).toEqual([ - { person: "c-1", key: wrapped("1", "kid-1"), hold: null }, - { person: "c-2", key: wrapped("2", "kid-2"), hold: null }, + { person: "c-1", category: D, key: wrapped("1", "kid-1"), hold: null }, + { person: "c-2", category: D, key: wrapped("2", "kid-2"), hold: null }, ]); expect(await store.listWrappedUnder({ kids: [null], limit: 10 })).toEqual([ - { person: "c-4", key: stored("4"), hold: null }, + { person: "c-4", category: D, key: stored("4"), hold: null }, ]); expect(await store.listWrappedUnder({ kids: [], limit: 10 })).toEqual([]); expect(await store.listWrappedUnder({ kids: ["kid-9"], limit: 10 })).toEqual([]); }); it("rewraps a key the person still holds, keeping its fingerprint", async () => { - await store.add("c-1", wrapped("first", "kid-1")); + await store.add("c-1", D, wrapped("first", "kid-1")); expect( await store.rewrap("c-1", wrapped("first", "kid-1"), { key: "w-first-kid-2", kid: "kid-2", }), ).toBe(true); - expect(await store.get("c-1")).toEqual(wrapped("first", "kid-2")); + expect(await store.get("c-1", D)).toEqual(wrapped("first", "kid-2")); // Another rewrap read the key before this one: it holds the same fingerprint, another text. expect( await store.rewrap("c-1", wrapped("first", "kid-1"), { key: "stale", kid: "kid-3" }), ).toBe(false); - expect(await store.get("c-1")).toEqual(wrapped("first", "kid-2")); + expect(await store.get("c-1", D)).toEqual(wrapped("first", "kid-2")); expect(await store.listWrappedUnder({ kids: ["kid-1"], limit: 10 })).toEqual([]); expect(await store.listWrappedUnder({ kids: ["kid-2"], limit: 10 })).toEqual([ - { person: "c-1", key: wrapped("first", "kid-2"), hold: null }, + { person: "c-1", category: D, key: wrapped("first", "kid-2"), hold: null }, ]); }); it("holds a key the person still holds, lists it again once the hold ends, and clears it with a rewrap", async () => { const first = wrapped("1", "kid-1"); - await store.add("c-1", first); - await store.add("c-2", wrapped("2", "kid-1")); - await store.add("c-3", stored("3")); + await store.add("c-1", D, first); + await store.add("c-2", D, wrapped("2", "kid-1")); + await store.add("c-3", D, stored("3")); const hold = { until: "2026-02-01T00:00:30.000Z", attempts: 2 }; await store.hold("c-1", first, hold); await store.hold("c-3", stored("3"), { until: "2026-02-01T00:00:20.000Z", attempts: 1 }); @@ -115,17 +125,17 @@ export const personalKeyStoreContract: PersonalKeyStoreContractFunction = ({ cre const end = new Date(hold.until); expect( await store.listWrappedUnder({ kids: ["kid-1"], releasedBy: before, limit: 10 }), - ).toEqual([{ person: "c-2", key: wrapped("2", "kid-1"), hold: null }]); + ).toEqual([{ person: "c-2", category: D, key: wrapped("2", "kid-1"), hold: null }]); expect(await store.listWrappedUnder({ kids: ["kid-1"], releasedBy: end, limit: 10 })).toEqual( [ - { person: "c-1", key: first, hold }, - { person: "c-2", key: wrapped("2", "kid-1"), hold: null }, + { person: "c-1", category: D, key: first, hold }, + { person: "c-2", category: D, key: wrapped("2", "kid-1"), hold: null }, ], ); expect(await store.listWrappedUnder({ kids: ["kid-1"], limit: 1 })).toEqual([ - { person: "c-1", key: first, hold }, + { person: "c-1", category: D, key: first, hold }, ]); - expect(await store.get("c-1")).toEqual(first); + expect(await store.get("c-1", D)).toEqual(first); expect(await store.nextHoldEnd({ kids: ["kid-1"], after: before })).toEqual(end); expect( await store.nextHoldEnd({ @@ -146,16 +156,16 @@ export const personalKeyStoreContract: PersonalKeyStoreContractFunction = ({ cre }); expect(await store.rewrap("c-1", first, { key: "w-1-kid-2", kid: "kid-2" })).toBe(true); expect(await store.listWrappedUnder({ kids: ["kid-2"], limit: 10 })).toEqual([ - { person: "c-1", key: wrapped("1", "kid-2"), hold: null }, + { person: "c-1", category: D, key: wrapped("1", "kid-2"), hold: null }, ]); expect(await store.nextHoldEnd({ kids: ["kid-1", "kid-2"], after: before })).toBeNull(); }); it("counts the keys wrapped under the ids asked for, held or not, and clears every hold at once", async () => { - await store.add("c-1", wrapped("1", "kid-1")); - await store.add("c-2", wrapped("2", "kid-1")); - await store.add("c-3", stored("3")); - await store.add("c-4", wrapped("4", "kid-2")); + await store.add("c-1", D, wrapped("1", "kid-1")); + await store.add("c-2", D, wrapped("2", "kid-1")); + await store.add("c-3", D, stored("3")); + await store.add("c-4", D, wrapped("4", "kid-2")); await store.hold("c-1", wrapped("1", "kid-1"), { until: "2026-02-01T00:00:30.000Z", attempts: 2, @@ -167,9 +177,9 @@ export const personalKeyStoreContract: PersonalKeyStoreContractFunction = ({ cre expect(await store.countWrappedUnder({ kids: [] })).toBe(0); expect(await store.releaseHolds()).toBe(2); expect(await store.listWrappedUnder({ kids: ["kid-1", null], limit: 10 })).toEqual([ - { person: "c-1", key: wrapped("1", "kid-1"), hold: null }, - { person: "c-2", key: wrapped("2", "kid-1"), hold: null }, - { person: "c-3", key: stored("3"), hold: null }, + { person: "c-1", category: D, key: wrapped("1", "kid-1"), hold: null }, + { person: "c-2", category: D, key: wrapped("2", "kid-1"), hold: null }, + { person: "c-3", category: D, key: stored("3"), hold: null }, ]); expect(await store.releaseHolds()).toBe(0); }); @@ -178,18 +188,18 @@ export const personalKeyStoreContract: PersonalKeyStoreContractFunction = ({ cre const read = wrapped("first", "kid-1"); const hold = { until: "2026-02-01T00:00:30.000Z", attempts: 1 }; await store.hold("c-1", read, hold); - expect(await store.get("c-1")).toBeNull(); - await store.add("c-1", wrapped("again", "kid-1")); + expect(await store.get("c-1", D)).toBeNull(); + await store.add("c-1", D, wrapped("again", "kid-1")); await store.hold("c-1", read, hold); await store.hold("c-1", { ...wrapped("again", "kid-1"), fingerprint: "f-other" }, hold); expect(await store.listWrappedUnder({ kids: ["kid-1"], limit: 10 })).toEqual([ - { person: "c-1", key: wrapped("again", "kid-1"), hold: null }, + { person: "c-1", category: D, key: wrapped("again", "kid-1"), hold: null }, ]); await store.hold("c-1", wrapped("again", "kid-1"), hold); - await store.remove("c-1"); - await store.add("c-1", wrapped("again", "kid-1")); + await store.remove("c-1", "f-again"); + await store.add("c-1", D, wrapped("again", "kid-1")); expect(await store.listWrappedUnder({ kids: ["kid-1"], limit: 10 })).toEqual([ - { person: "c-1", key: wrapped("again", "kid-1"), hold: null }, + { person: "c-1", category: D, key: wrapped("again", "kid-1"), hold: null }, ]); }); @@ -197,17 +207,254 @@ export const personalKeyStoreContract: PersonalKeyStoreContractFunction = ({ cre const read = wrapped("first", "kid-1"); const to = { key: "w-first-kid-2", kid: "kid-2" }; expect(await store.rewrap("c-1", read, to)).toBe(false); - expect(await store.get("c-1")).toBeNull(); - await store.add("c-1", read); - await store.remove("c-1"); + expect(await store.get("c-1", D)).toBeNull(); + await store.add("c-1", D, read); + await store.remove("c-1", "f-first"); expect(await store.rewrap("c-1", read, to)).toBe(false); - expect(await store.get("c-1")).toBeNull(); - await store.add("c-1", wrapped("again", "kid-1")); + expect(await store.get("c-1", D)).toBeNull(); + await store.add("c-1", D, wrapped("again", "kid-1")); expect(await store.rewrap("c-1", read, to)).toBe(false); - expect(await store.get("c-1")).toEqual(wrapped("again", "kid-1")); - await store.add("c-2", read); + expect(await store.get("c-1", D)).toEqual(wrapped("again", "kid-1")); + await store.add("c-2", D, read); expect(await store.rewrap("c-2", { ...read, fingerprint: "f-other" }, to)).toBe(false); - expect(await store.get("c-2")).toEqual(read); + expect(await store.get("c-2", D)).toEqual(read); + }); + + it("keeps a key in force per person and category, each found by its fingerprint", async () => { + expect(await store.add("c-1", D, stored("1"))).toEqual(stored("1")); + expect(await store.add("c-1", "invoices", stored("2"))).toEqual(stored("2")); + expect(await store.add("c-1", "invoices", stored("3"))).toEqual(stored("2")); + expect(await store.get("c-1", D)).toEqual(stored("1")); + expect(await store.get("c-1", "invoices")).toEqual(stored("2")); + expect(await store.get("c-1", "profile")).toBeNull(); + expect(await store.find("c-1", "f-2")).toEqual({ + category: "invoices", + key: stored("2"), + retainedSince: null, + lastActiveAt: null, + }); + expect(await store.find("c-2", "f-2")).toBeNull(); + expect((await store.list("c-1")).map(({ category }) => category)).toEqual([D, "invoices"]); + expect(await store.remove("c-1", "f-2")).toBe(true); + expect(await store.get("c-1", "invoices")).toBeNull(); + expect(await store.get("c-1", D)).toEqual(stored("1")); + }); + + it("starts a new key from the person's last activity, unless they were forgotten since", async () => { + await persons.record({ + person: "c-1", + position: 1, + lastActiveAt: "2026-01-01T00:00:00.000Z", + forgottenAt: null, + }); + await persons.record({ + person: "c-2", + position: 2, + lastActiveAt: "2026-01-01T00:00:00.000Z", + forgottenAt: null, + }); + await persons.record({ person: "c-2", position: 3, forgottenAt: "2026-01-02T00:00:00.000Z" }); + await store.add("c-1", "profile", stored("1")); + await store.add("c-2", "profile", stored("2")); + await store.add("c-3", "profile", stored("3")); + expect((await store.find("c-1", "f-1"))?.lastActiveAt).toBe("2026-01-01T00:00:00.000Z"); + expect((await store.find("c-2", "f-2"))?.lastActiveAt).toBeNull(); + expect((await store.find("c-3", "f-3"))?.lastActiveAt).toBeNull(); + }); + + it("retains a key in force past a forgetting, keeps it findable, and takes a new one for its category", async () => { + await store.add("c-1", "invoices", stored("old")); + const kept = { since: "2026-02-01T00:00:00.000Z", lastActiveAt: "2026-01-15T00:00:00.000Z" }; + expect(await store.retain("c-1", "f-old", kept)).toBe(true); + expect(await store.retain("c-1", "f-old", kept)).toBe(false); + expect(await store.retain("c-1", "f-none", kept)).toBe(false); + expect(await store.get("c-1", "invoices")).toBeNull(); + expect(await store.find("c-1", "f-old")).toEqual({ + category: "invoices", + key: stored("old"), + retainedSince: kept.since, + lastActiveAt: kept.lastActiveAt, + }); + expect(await store.add("c-1", "invoices", stored("new"))).toEqual(stored("new")); + expect(await store.get("c-1", "invoices")).toEqual(stored("new")); + expect((await store.list("c-1")).map(({ key }) => key.fingerprint)).toEqual([ + "f-new", + "f-old", + ]); + }); + + it("records the last activity on the keys in force, never moving it back", async () => { + await store.add("c-1", D, stored("1")); + await store.add("c-1", "invoices", stored("2")); + await store.retain("c-1", "f-2", { + since: "2026-02-01T00:00:00.000Z", + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + await store.touch("c-1", "2026-03-01T00:00:00.000Z"); + await store.touch("c-1", "2026-02-01T00:00:00.000Z"); + expect((await store.find("c-1", "f-1"))?.lastActiveAt).toBe("2026-03-01T00:00:00.000Z"); + expect((await store.find("c-1", "f-2"))?.lastActiveAt).toBe("2026-01-01T00:00:00.000Z"); + }); + + it("lists the keys of a category by last activity, kept ones too, leaving out the restricted and the never active", async () => { + for (const person of ["c-1", "c-2", "c-3", "c-4", "c-6"]) { + await store.add(person, "profile", stored(person.slice(2))); + } + await store.add("c-5", D, stored("5")); + await store.touch("c-1", "2026-01-03T00:00:00.000Z"); + await store.touch("c-2", "2026-01-01T00:00:00.000Z"); + await store.touch("c-3", "2026-01-02T00:00:00.000Z"); + await store.touch("c-5", "2026-01-01T00:00:00.000Z"); + await store.touch("c-6", "2026-01-01T00:00:00.000Z"); + await store.retain("c-3", "f-3", { + since: "2026-01-04T00:00:00.000Z", + lastActiveAt: "2026-01-02T00:00:00.000Z", + }); + await persons.restrict("c-6", new Date("2026-01-05T00:00:00.000Z")); + const due = (person: string, lastActiveAt: string, retainedSince: string | null = null) => ({ + person, + category: "profile", + fingerprint: `f-${person.slice(2)}`, + lastActiveAt, + retainedSince, + hold: null, + }); + expect(await store.listDue({ category: "profile", limit: 10 })).toEqual([ + due("c-2", "2026-01-01T00:00:00.000Z"), + due("c-3", "2026-01-02T00:00:00.000Z", "2026-01-04T00:00:00.000Z"), + due("c-1", "2026-01-03T00:00:00.000Z"), + ]); + expect( + await store.listDue({ + category: "profile", + activeBefore: new Date("2026-01-02T00:00:00.000Z"), + limit: 10, + }), + ).toEqual([ + due("c-2", "2026-01-01T00:00:00.000Z"), + due("c-3", "2026-01-02T00:00:00.000Z", "2026-01-04T00:00:00.000Z"), + ]); + expect(await store.listDue({ category: "profile", limit: 1 })).toEqual([ + due("c-2", "2026-01-01T00:00:00.000Z"), + ]); + expect(await store.listDue({ category: "invoices", limit: 10 })).toEqual([]); + }); + + it("holds a due key only while it records the activity it was listed with, and touching it again clears the hold", async () => { + await store.add("c-1", "profile", stored("1")); + await store.add("c-2", "profile", stored("2")); + await store.touch("c-1", "2026-01-01T00:00:00.000Z"); + await store.touch("c-2", "2026-01-02T00:00:00.000Z"); + const hold = { until: "2026-02-01T00:00:30.000Z", attempts: 2 }; + await store.holdDue("c-1", "f-1", "2026-01-01T00:00:00.000Z", hold); + await store.holdDue("c-2", "f-2", "2025-12-31T00:00:00.000Z", hold); + const before = new Date("2026-02-01T00:00:29.999Z"); + const end = new Date(hold.until); + expect( + (await store.listDue({ category: "profile", releasedBy: before, limit: 10 })).map( + ({ person }) => person, + ), + ).toEqual(["c-2"]); + expect( + (await store.listDue({ category: "profile", releasedBy: end, limit: 10 })).map( + ({ person, hold: held }) => [person, held], + ), + ).toEqual([ + ["c-1", hold], + ["c-2", null], + ]); + expect(await store.nextDueHoldEnd({ category: "profile", after: before })).toEqual(end); + expect(await store.nextDueHoldEnd({ category: "profile", after: end })).toBeNull(); + expect(await store.nextDueHoldEnd({ category: "invoices", after: before })).toBeNull(); + await store.touch("c-1", "2026-01-05T00:00:00.000Z"); + expect(await store.nextDueHoldEnd({ category: "profile", after: before })).toBeNull(); + }); + + it("lists the keys a forgetting kept outside the categories given, kept longest first, leaving out the restricted", async () => { + const kept = async (person: string, category: string, since: string) => { + await store.add(person, category, stored(person.slice(2))); + await store.retain(person, `f-${person.slice(2)}`, { + since, + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + }; + await kept("c-1", "invoices", "2026-02-03T00:00:00.000Z"); + await kept("c-2", "billing", "2026-02-02T00:00:00.000Z"); + await kept("c-3", "orders", "2026-02-01T00:00:00.000Z"); + await kept("c-4", "billing", "2026-02-01T00:00:00.000Z"); + await store.add("c-5", "billing", stored("5")); + await persons.restrict("c-4", new Date("2026-02-05T00:00:00.000Z")); + const listed = await store.listRetained({ except: ["invoices"], limit: 10 }); + expect(listed.map(({ person }) => person)).toEqual(["c-3", "c-2"]); + expect(listed[0]).toEqual({ + person: "c-3", + category: "orders", + fingerprint: "f-3", + lastActiveAt: "2026-01-01T00:00:00.000Z", + retainedSince: "2026-02-01T00:00:00.000Z", + hold: null, + }); + expect( + (await store.listRetained({ except: [], limit: 10 })).map(({ person }) => person), + ).toEqual(["c-3", "c-2", "c-1"]); + expect(await store.listRetained({ except: ["invoices"], limit: 1 })).toHaveLength(1); + const hold = { until: "2026-03-01T00:00:00.000Z", attempts: 1 }; + await store.holdDue("c-3", "f-3", "2026-01-01T00:00:00.000Z", hold); + const before = new Date("2026-02-28T00:00:00.000Z"); + expect( + (await store.listRetained({ except: ["invoices"], releasedBy: before, limit: 10 })).map( + ({ person }) => person, + ), + ).toEqual(["c-2"]); + expect(await store.nextRetainedHoldEnd({ except: ["invoices"], after: before })).toEqual( + new Date(hold.until), + ); + expect(await store.nextRetainedHoldEnd({ except: ["orders"], after: before })).toBeNull(); + expect( + await store.nextRetainedHoldEnd({ except: [], after: new Date(hold.until) }), + ).toBeNull(); + }); + + it("counts what the retention lists, held or not, and clears the retention's holds alone", async () => { + for (const person of ["c-1", "c-2", "c-3", "c-4"]) { + await store.add(person, "profile", stored(person.slice(2))); + await store.touch(person, `2026-01-0${person.slice(2)}T00:00:00.000Z`); + } + await store.add("c-5", "billing", stored("5")); + await store.retain("c-5", "f-5", { + since: "2026-02-01T00:00:00.000Z", + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + await store.add("c-6", "invoices", stored("6")); + await store.retain("c-6", "f-6", { + since: "2026-02-01T00:00:00.000Z", + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + await persons.restrict("c-4", new Date("2026-02-05T00:00:00.000Z")); + const hold = { until: "2026-03-01T00:00:00.000Z", attempts: 1 }; + await store.holdDue("c-1", "f-1", "2026-01-01T00:00:00.000Z", hold); + await store.holdDue("c-5", "f-5", "2026-01-01T00:00:00.000Z", hold); + await store.hold("c-2", stored("2"), hold); + expect(await store.countDue({ category: "profile" })).toBe(3); + expect( + await store.countDue({ + category: "profile", + activeBefore: new Date("2026-01-02T00:00:00.000Z"), + }), + ).toBe(2); + expect(await store.countDue({ category: "billing" })).toBe(1); + expect(await store.countRetained({ except: ["invoices"] })).toBe(1); + expect(await store.countRetained({ except: ["invoices", "billing"] })).toBe(0); + expect(await store.countRetained({ except: [] })).toBe(2); + expect(await store.releaseDueHolds()).toBe(2); + expect(await store.releaseDueHolds()).toBe(0); + expect( + (await store.listDue({ category: "profile", limit: 10 })).map(({ hold: held }) => held), + ).toEqual([null, null, null]); + expect( + await store.listWrappedUnder({ kids: [null], releasedBy: new Date(0), limit: 10 }), + ).toHaveLength(5); + expect(await store.releaseHolds()).toBe(1); }); }); }; diff --git a/packages/core/src/adapter/testing/scheduler.contract.ts b/packages/core/src/adapter/testing/scheduler.contract.ts index 26dd56ca..67da35c8 100644 --- a/packages/core/src/adapter/testing/scheduler.contract.ts +++ b/packages/core/src/adapter/testing/scheduler.contract.ts @@ -1,5 +1,6 @@ import { beforeEach, describe, expect, it } from "vitest"; import { ScheduledClaimLostError } from "../../contracts/errors.ts"; +import type { KeyedPerson } from "../storage/personal-key-store.ts"; import type { ClaimedCommand, ScheduleArgs, Scheduler } from "../storage/scheduler.ts"; import { testKeyedPerson as held, testCommand, testContext, testSealedValue } from "./fixtures.ts"; @@ -594,6 +595,25 @@ export const schedulerContract: SchedulerContractFunction = ({ create }) => { expect(await scheduler.listByPerson("c-1")).toEqual([]); }); + it("forgets the commands that name a person under a key outside those kept, or in the clear", async () => { + const schedule = (dedupeKey: string, persons: readonly KeyedPerson[]) => + scheduler.schedule({ + dedupeKey, + command: testCommand(dedupeKey), + executeAt: at(1_000), + context: testContext, + persons, + }); + await schedule("a", [held("c-1", "1")]); + await schedule("b", [held("c-1", "2")]); + await schedule("c", [held("c-1", "1"), held("c-1", "2")]); + await schedule("d", [held("c-1", null)]); + await schedule("e", [held("c-2", "3")]); + const forgotten = await scheduler.forget("c-1", ["f-2"]); + expect(forgotten.map((entry) => entry.dedupeKey)).toEqual(["a", "c", "d"]); + expect((await scheduler.list()).map((entry) => entry.dedupeKey)).toEqual(["b", "e"]); + }); + it("replaces whose personal data a command holds when its key is scheduled again", async () => { const schedule = (persons?: ScheduleArgs["persons"]) => scheduler.schedule({ diff --git a/packages/core/src/adapter/testing/snapshot-store.contract.ts b/packages/core/src/adapter/testing/snapshot-store.contract.ts index cb33e457..c2e166db 100644 --- a/packages/core/src/adapter/testing/snapshot-store.contract.ts +++ b/packages/core/src/adapter/testing/snapshot-store.contract.ts @@ -1,6 +1,7 @@ import { beforeEach, describe, expect, it } from "vitest"; import type { Storage } from "../adapter.ts"; import type { KeyedPerson, PersonalKeyStore } from "../storage/personal-key-store.ts"; +import { DEFAULT_PERSONAL_CATEGORY as D } from "../storage/personal-key-store.ts"; import type { Snapshot, SnapshotStore } from "../storage/snapshot-store.ts"; import { testKeyedPerson as held, testPersonalKey as stored } from "./fixtures.ts"; @@ -82,13 +83,13 @@ export const snapshotStoreContract: SnapshotStoreContractFunction = ({ create }) const ada: Snapshot = { ...order42, persons: [held("c-1", "1"), held("c-2", "2")] }; await store.put(ada); expect(await store.get(order42)).toBeNull(); - await keys.add("c-1", stored("1")); + await keys.add("c-1", D, stored("1")); await store.put(ada); expect(await store.get(order42)).toBeNull(); - await keys.add("c-2", stored("2")); + await keys.add("c-2", D, stored("2")); await store.put(ada); expect(await store.get(order42)).toEqual(ada); - await keys.remove("c-2"); + await keys.remove("c-2", "f-2"); await store.put({ ...ada, version: 300 }); expect(await store.get(order42)).toEqual(ada); }); @@ -104,29 +105,29 @@ export const snapshotStoreContract: SnapshotStoreContractFunction = ({ create }) }; await store.put(mixed); expect(await store.get(order42)).toEqual(clear); - await keys.add("c-2", stored("2")); + await keys.add("c-2", D, stored("2")); await store.put(mixed); expect(await store.get(order42)).toEqual(mixed); }); it("writes nothing for a person named with a key other than the one they hold", async () => { const named = (...persons: KeyedPerson[]): Snapshot => ({ ...order42, persons }); - await keys.add("c-1", stored("new")); + await keys.add("c-1", D, stored("new")); await store.put(named(held("c-1", "old"))); expect(await store.get(order42)).toBeNull(); await store.put(named(held("c-1", "old"), held("c-1", "new"))); expect(await store.get(order42)).toBeNull(); await store.put(named(held("c-1", "new"), held("c-1", null))); expect(await store.get(order42)).toEqual(named(held("c-1", "new"), held("c-1", null))); - await keys.remove("c-1"); - await keys.add("c-1", stored("newer")); + await keys.remove("c-1", "f-new"); + await keys.add("c-1", D, stored("newer")); await store.put({ ...named(held("c-1", "new")), version: 300 }); expect((await store.get(order42))?.version).toBe(order42.version); }); it("forgets every snapshot that names a person, and no other", async () => { - await keys.add("c-1", stored("1")); - await keys.add("c-2", stored("2")); + await keys.add("c-1", D, stored("1")); + await keys.add("c-2", D, stored("2")); const c1 = held("c-1", "1"); const c2 = held("c-2", "2"); const named = (aggregateId: string, persons: readonly KeyedPerson[]): Snapshot => ({ @@ -150,6 +151,37 @@ export const snapshotStoreContract: SnapshotStoreContractFunction = ({ create }) expect(await store.get({ aggregateType: "customer", aggregateId: "1" })).toBeNull(); }); + it("forgets the snapshots that name a person under a key outside those kept, or in the clear", async () => { + const named = (aggregateId: string, persons: readonly KeyedPerson[]): Snapshot => ({ + ...order42, + aggregateId, + persons, + }); + await keys.add("c-1", D, stored("1")); + await keys.add("c-1", "invoices", stored("2")); + await store.put(named("1", [held("c-1", "1")])); + await store.put(named("2", [held("c-1", "2")])); + await store.put(named("3", [held("c-1", "1"), held("c-1", "2")])); + await store.put(named("4", [held("c-1", null)])); + await store.forget("c-1", ["f-2"]); + const left = await Promise.all( + ["1", "2", "3", "4"].map(async (aggregateId) => + (await store.get({ aggregateType: "order", aggregateId })) === null ? null : aggregateId, + ), + ); + expect(left).toEqual([null, "2", null, null]); + }); + + it("writes nothing for a person whose key was kept past a forgetting", async () => { + await keys.add("c-1", "invoices", stored("1")); + await keys.retain("c-1", "f-1", { + since: "2026-02-01T00:00:00.000Z", + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + await store.put({ ...order42, persons: [held("c-1", "1")] }); + expect(await store.get(order42)).toBeNull(); + }); + it("forgets the snapshot of the stream it is asked to remove, and only that one", async () => { await store.put(order42); await store.put({ ...order42, aggregateId: "43" }); diff --git a/packages/core/src/adapter/testing/storage-schema.contract.ts b/packages/core/src/adapter/testing/storage-schema.contract.ts index d5b41246..397a1465 100644 --- a/packages/core/src/adapter/testing/storage-schema.contract.ts +++ b/packages/core/src/adapter/testing/storage-schema.contract.ts @@ -35,6 +35,7 @@ const LATEST = [ { scope: "checkpoints", version: 1 }, { scope: "storage", version: 1 }, { scope: "storage", version: 2 }, + { scope: "storage", version: 3 }, ]; const versionsOf = async ({ execute, prefix }: StorageSchemaTarget) => @@ -147,20 +148,20 @@ export const storageSchemaContract: StorageSchemaContractFunction = ({ const target = await create(); const versions = await target.adapter.schemaVersions(); expect(versions).toEqual([ - { scope: "storage", version: 0, latest: 2 }, + { scope: "storage", version: 0, latest: 3 }, { scope: "checkpoints", version: 0, latest: 1 }, - { scope: "forget-ledger", version: 0, latest: 3 }, + { scope: "forget-ledger", version: 0, latest: 4 }, ]); expect(await shapeOf(target, dialect)).toEqual([]); expect(await target.adapter.migrate(["checkpoints"])).toEqual([ - { scope: "storage", version: 0, latest: 2 }, + { scope: "storage", version: 0, latest: 3 }, { scope: "checkpoints", version: 1, latest: 1 }, - { scope: "forget-ledger", version: 0, latest: 3 }, + { scope: "forget-ledger", version: 0, latest: 4 }, ]); expect(await target.adapter.migrate(["storage", "forget-ledger"])).toEqual([ - { scope: "storage", version: 2, latest: 2 }, + { scope: "storage", version: 3, latest: 3 }, { scope: "checkpoints", version: 1, latest: 1 }, - { scope: "forget-ledger", version: 3, latest: 3 }, + { scope: "forget-ledger", version: 4, latest: 4 }, ]); const shape = await shapeOf(target, dialect); await opened(target.adapter, async () => undefined); @@ -214,7 +215,7 @@ export const storageSchemaContract: StorageSchemaContractFunction = ({ code: "STORAGE_SCHEMA_TOO_NEW", scope: "storage", version: 999, - supported: 2, + supported: 3, }); expect(await shapeOf(target, dialect)).toEqual(shape); }); @@ -304,6 +305,7 @@ export const storageSchemaContract: StorageSchemaContractFunction = ({ personId: "p-1", eventId: "e-1", forgottenAt: "2026-01-01T00:00:00.000Z", + categories: null, reason: "requested", }, ]); @@ -314,6 +316,7 @@ export const storageSchemaContract: StorageSchemaContractFunction = ({ { scope: "forget-ledger", version: 1 }, { scope: "forget-ledger", version: 2 }, { scope: "forget-ledger", version: 3 }, + { scope: "forget-ledger", version: 4 }, ]); }); diff --git a/packages/core/src/adapter/testing/storage-transaction.contract.ts b/packages/core/src/adapter/testing/storage-transaction.contract.ts index 6dcc9401..7bddd947 100644 --- a/packages/core/src/adapter/testing/storage-transaction.contract.ts +++ b/packages/core/src/adapter/testing/storage-transaction.contract.ts @@ -7,6 +7,7 @@ import { } from "../../contracts/errors.ts"; import type { Storage, StorageTransaction } from "../adapter.ts"; import type { KeyedPerson } from "../storage/personal-key-store.ts"; +import { DEFAULT_PERSONAL_CATEGORY as D } from "../storage/personal-key-store.ts"; import { testKeyedPerson as held, pendingEvent, @@ -143,7 +144,7 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ await expect( storage.transact(async (tx) => { await tx.inboxLedger.complete(key); - await tx.personalKeys.add("c-9", stored("9")); + await tx.personalKeys.add("c-9", D, stored("9")); await tx.scheduler.schedule({ dedupeKey: "command:c1", command: testCommand("1"), @@ -159,7 +160,7 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ }), ).rejects.toBeInstanceOf(ConcurrencyError); await nothingWritten(); - expect(await storage.personalKeys.get("c-9")).toBeNull(); + expect(await storage.personalKeys.get("c-9", D)).toBeNull(); }); it("writes nothing when the scheduler entry it settles was claimed again", async () => { @@ -274,11 +275,11 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ }); it("adds and removes personal keys with the rest, or not at all", async () => { - await storage.personalKeys.add("c-1", stored("1")); + await storage.personalKeys.add("c-1", D, stored("1")); const work = async (tx: StorageTransaction) => { - expect(await tx.personalKeys.remove("c-1")).toBe(true); - expect(await tx.personalKeys.add("c-2", stored("2"))).toEqual(stored("2")); - expect(await tx.personalKeys.get("c-2")).toEqual(stored("2")); + expect(await tx.personalKeys.remove("c-1", "f-1")).toBe(true); + expect(await tx.personalKeys.add("c-2", D, stored("2"))).toEqual(stored("2")); + expect(await tx.personalKeys.get("c-2", D)).toEqual(stored("2")); await tx.eventStore.append({ aggregateType: "order", aggregateId: "1", @@ -292,12 +293,12 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ throw new Error("boom"); }), ).rejects.toThrow("boom"); - expect(await storage.personalKeys.get("c-1")).toEqual(stored("1")); - expect(await storage.personalKeys.get("c-2")).toBeNull(); + expect(await storage.personalKeys.get("c-1", D)).toEqual(stored("1")); + expect(await storage.personalKeys.get("c-2", D)).toBeNull(); expect(await storage.eventStore.lastPosition()).toBe(0); await storage.transact(work); - expect(await storage.personalKeys.get("c-1")).toBeNull(); - expect(await storage.personalKeys.get("c-2")).toEqual(stored("2")); + expect(await storage.personalKeys.get("c-1", D)).toBeNull(); + expect(await storage.personalKeys.get("c-2", D)).toEqual(stored("2")); expect(await storage.eventStore.readAll({ afterPosition: 0, limit: 10 })).toHaveLength(1); }); @@ -363,6 +364,7 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ person: "c-1", restrictedSince: null, lastActiveAt: now.toISOString(), + position: 3, forgottenAt: null, hold: null, }, @@ -371,7 +373,7 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ it("holds a person and a key with the rest, or not at all", async () => { const key = stored("1"); - await storage.personalKeys.add("c-1", key); + await storage.personalKeys.add("c-1", D, key); await storage.persons.record({ person: "c-1", position: 1, @@ -388,7 +390,7 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ ).rejects.toThrow("boom"); expect((await storage.persons.get("c-1"))?.hold).toBeNull(); expect(await storage.personalKeys.listWrappedUnder({ kids: [null], limit: 10 })).toEqual([ - { person: "c-1", key, hold: null }, + { person: "c-1", category: D, key, hold: null }, ]); await storage.transact(async (tx) => { await tx.persons.hold("c-1", now.toISOString(), hold); @@ -404,13 +406,13 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ }); expect((await storage.persons.get("c-1"))?.hold).toEqual(hold); expect(await storage.personalKeys.listWrappedUnder({ kids: [null], limit: 10 })).toEqual([ - { person: "c-1", key, hold }, + { person: "c-1", category: D, key, hold }, ]); }); it("forgets a person's snapshots and remembered rejections with their key", async () => { - await storage.personalKeys.add("c-1", stored("1")); - await storage.personalKeys.add("c-2", stored("2")); + await storage.personalKeys.add("c-1", D, stored("1")); + await storage.personalKeys.add("c-2", D, stored("2")); const snapshot = (aggregateId: string, persons: readonly KeyedPerson[]) => ({ aggregateType: "customer", aggregateId, @@ -436,12 +438,12 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ }; await storage.idempotencyStore.put(remembered); await storage.transact(async (tx) => { - await tx.personalKeys.remove("c-1"); + await tx.personalKeys.remove("c-1", "f-1"); await tx.snapshotStore.forget("c-1"); await tx.idempotencyStore.forget("c-1"); }); expect(await storage.idempotencyStore.get({ key: "req-1", now })).toBeNull(); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", D)).toBeNull(); expect(await storage.snapshotStore.get(snapshot("c-1", []))).toBeNull(); expect(await storage.snapshotStore.get(snapshot("c-2", []))).toEqual( snapshot("c-2", [held("c-2", "2")]), @@ -481,7 +483,7 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ }); it("commits events naming a person the same transaction forgets or restricts after appending them", async () => { - await storage.personalKeys.add("c-1", stored("one")); + await storage.personalKeys.add("c-1", D, stored("one")); await storage.transact(async (tx) => { await tx.eventStore.append({ aggregateType: "order", @@ -491,11 +493,11 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ persons: [held("c-1", "one")], touches: ["c-1"], }); - await tx.personalKeys.remove("c-1"); + await tx.personalKeys.remove("c-1", "f-one"); await tx.persons.restrict("c-1", now); }); expect(await storage.eventStore.lastPosition()).toBe(1); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", D)).toBeNull(); expect((await storage.persons.get("c-1"))?.restrictedSince).not.toBeNull(); }); @@ -612,7 +614,7 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ await storage.deadLetterStore.add(testDeadLetter("d1")); const sealed = { $personal: { person: "c-1", key: "f-1", iv: "iv", data: "data" } }; const reseal = async (tx: StorageTransaction) => { - await tx.personalKeys.add("c-1", stored("1")); + await tx.personalKeys.add("c-1", D, stored("1")); await tx.eventStore.reseal([{ position: 1, payload: { email: sealed }, person: "c-1" }]); await tx.scheduler.reseal({ dedupeKey: "command:c1", @@ -628,7 +630,7 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ throw new Error("boom"); }), ).rejects.toThrow("boom"); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", D)).toBeNull(); expect( await storage.eventStore.readByPerson({ person: "c-1", afterPosition: 0, limit: 10 }), ).toEqual([]); @@ -639,7 +641,7 @@ export const storageTransactionContract: StorageTransactionContractFunction = ({ }); await storage.transact(reseal); - expect(await storage.personalKeys.get("c-1")).toEqual(stored("1")); + expect(await storage.personalKeys.get("c-1", D)).toEqual(stored("1")); const [event] = await storage.eventStore.readByPerson({ person: "c-1", afterPosition: 0, diff --git a/packages/core/src/config/schema.test.ts b/packages/core/src/config/schema.test.ts index 46cbbfc2..859eda41 100644 --- a/packages/core/src/config/schema.test.ts +++ b/packages/core/src/config/schema.test.ts @@ -67,7 +67,7 @@ describe("resolveConfig", () => { backoff: { baseDelayMs: 1_000, maxDelayMs: 30_000 }, catchUp: { timeoutMs: 2_000, pollIntervalMs: 15 }, }, - personalData: { retentionMs: null, keyWrapper: null, forgetLedger: null }, + personalData: { retention: null, keyWrapper: null, forgetLedger: null }, deadLetters: { retentionMs: null }, telemetry: { errorText: "omit" }, overrides: {}, @@ -130,7 +130,7 @@ describe("resolveConfig", () => { catchUp: { timeoutMs: 5_000, pollIntervalMs: 50 }, }); expect(resolved.runtime.personalData).toEqual({ - retentionMs: 2_592_000_000, + retention: { months: 0, ms: 2_592_000_000 }, keyWrapper, forgetLedger: { store: forgetLedger, @@ -272,8 +272,18 @@ describe("resolveConfig", () => { message({ storage: sqlite, runtime: { processes: { handlerTimeout: "0s" } } }), ).toContain("runtime.processes.handlerTimeout: Expected a duration longer than 0"); expect(message({ storage: sqlite, runtime: { personalData: { retention: 0 } } })).toContain( - "runtime.personalData.retention: Expected a duration longer than 0", + "runtime.personalData.retention: Expected a retention longer than 0", ); + expect( + message({ storage: sqlite, runtime: { personalData: { retention: "1.5y" as "1y" } } }), + ).toContain("runtime.personalData.retention: Invalid retention"); + expect(message({ storage: sqlite, runtime: { personalData: { retention: "0mo" } } })).toContain( + "runtime.personalData.retention: Expected a retention longer than 0", + ); + expect( + resolveConfig({ storage: sqlite, runtime: { personalData: { retention: "1mo" } } }).runtime + .personalData.retention, + ).toEqual({ months: 1, ms: 0 }); expect(message({ storage: sqlite, runtime: { deadLetters: { retention: 0 } } })).toContain( "runtime.deadLetters.retention: Expected a duration longer than 0", ); diff --git a/packages/core/src/config/schema.ts b/packages/core/src/config/schema.ts index 1219b61d..579e8749 100644 --- a/packages/core/src/config/schema.ts +++ b/packages/core/src/config/schema.ts @@ -1,6 +1,6 @@ import { z } from "zod"; import { isAdapterDefinition } from "../adapter/adapter-definition.ts"; -import { parseDuration } from "../contracts/duration.ts"; +import { parseDuration, parseRetention } from "../contracts/duration.ts"; import { ConfigurationError } from "../contracts/errors.ts"; import { type ForgetLedgerSource, @@ -54,6 +54,17 @@ const timeout = duration.refine((milliseconds) => milliseconds > 0, { message: "Expected a duration longer than 0", }); +const retention = z.union([z.number(), z.string()]).transform((value, context) => { + try { + const parsed = parseRetention(value); + if (parsed.months > 0 || parsed.ms > 0) return parsed; + context.addIssue({ code: "custom", message: "Expected a retention longer than 0" }); + } catch (error) { + context.addIssue({ code: "custom", message: (error as Error).message }); + } + return z.NEVER; +}); + const adapter = z.custom(isAdapterDefinition, { message: "Expected an adapter definition such as sqlite({ ... })", }); @@ -141,7 +152,7 @@ const runtime = z.strictObject({ .optional(), personalData: z .strictObject({ - retention: timeout.optional(), + retention: retention.optional(), keyWrapper: z .custom(isKeyWrapperDefinition, { message: 'Expected a key wrapper such as masterKey({ from: "BOUNDA_MASTER_KEY" })', @@ -297,7 +308,7 @@ export const resolveConfig: ResolveConfigFunction = (config) => { }, }, personalData: { - retentionMs: parsed.runtime?.personalData?.retention ?? null, + retention: parsed.runtime?.personalData?.retention ?? null, keyWrapper: parsed.runtime?.personalData?.keyWrapper ?? null, forgetLedger: resolveForgetLedger(parsed.runtime?.personalData?.forgetLedger), }, diff --git a/packages/core/src/config/types.ts b/packages/core/src/config/types.ts index 45184718..f1014f13 100644 --- a/packages/core/src/config/types.ts +++ b/packages/core/src/config/types.ts @@ -1,5 +1,5 @@ import type { AdapterDefinition } from "../adapter/adapter-definition.ts"; -import type { DurationInput } from "../contracts/duration.ts"; +import type { DurationInput, Retention, RetentionInput } from "../contracts/duration.ts"; import type { ForgetLedgerSource } from "../kernel/personal-data/forget-ledger.ts"; import type { KeyWrapperDefinition } from "../kernel/personal-data/key-wrapper.ts"; import type { AppPortsConfig } from "../register/index.ts"; @@ -135,9 +135,10 @@ export interface PersonalDataConfig { * How long a person may go without an event of the app's aggregates naming them before the * runtime forgets them, as `forget` does, with `reason: "retention"`. A person whose processing * is restricted is left alone until it is lifted, and one active again after being forgotten - * starts a new clock. No default: without it nobody is forgotten for inactivity. + * starts a new clock. A duration, or whole calendar months or years, `"6y"`, as a category's + * retention. No default: without it nobody is forgotten for inactivity. */ - readonly retention?: DurationInput; + readonly retention?: RetentionInput; /** * Wraps every person's key before the store keeps it, so a copy of the database holds no key * that opens their data: `masterKey({ from: "BOUNDA_MASTER_KEY" })`, or a wrapper of your own @@ -412,7 +413,7 @@ export interface ResolvedConfig { /** * `null` when no retention is configured. */ - readonly retentionMs: number | null; + readonly retention: Retention | null; /** * `null` when no key wrapper is configured. */ diff --git a/packages/core/src/contracts/duration.test.ts b/packages/core/src/contracts/duration.test.ts index 6c9e1750..53d80b05 100644 --- a/packages/core/src/contracts/duration.test.ts +++ b/packages/core/src/contracts/duration.test.ts @@ -1,5 +1,11 @@ import { describe, expect, it } from "vitest"; -import { asDuration, parseDuration } from "./duration.ts"; +import { + asDuration, + parseDuration, + parseRetention, + retentionEnd, + retentionStart, +} from "./duration.ts"; import { ValidationError } from "./errors.ts"; describe("parseDuration", () => { @@ -54,3 +60,79 @@ describe("asDuration", () => { expect(() => asDuration("soon")).toThrow(ValidationError); }); }); + +describe("parseRetention", () => { + it("takes a duration, or whole calendar months and years", () => { + expect(parseRetention("30d")).toEqual({ months: 0, ms: 2_592_000_000 }); + expect(parseRetention(1_000)).toEqual({ months: 0, ms: 1_000 }); + expect(parseRetention("6y")).toEqual({ months: 72, ms: 0 }); + expect(parseRetention("18mo")).toEqual({ months: 18, ms: 0 }); + expect(parseRetention("5m")).toEqual({ months: 0, ms: 300_000 }); + }); + + it("refuses a fraction of a month or a year, and anything that is not a duration", () => { + for (const input of ["1.5y", "0.5mo"]) { + expect(() => parseRetention(input)).toThrow( + new ValidationError(`Invalid retention: "${input}"`, [ + { path: [], message: "Expected a whole number of months or years" }, + ]), + ); + } + expect(() => parseRetention("6 years")).toThrow(ValidationError); + expect(() => parseRetention("y")).toThrow(ValidationError); + }); +}); + +describe("retentionEnd and retentionStart", () => { + const at = (text: string): number => Date.parse(text); + const iso = (time: number): string => new Date(time).toISOString(); + const years = (count: number) => ({ months: count * 12, ms: 0 }); + const months = (count: number) => ({ months: count, ms: 0 }); + + it("ends a calendar period on the same day and time, whatever the leap years between", () => { + expect(iso(retentionEnd(at("2026-03-15T10:30:00.000Z"), years(6)))).toBe( + "2032-03-15T10:30:00.000Z", + ); + expect(iso(retentionEnd(at("2026-01-31T08:00:00.000Z"), months(1)))).toBe( + "2026-02-28T23:59:59.999Z", + ); + expect(iso(retentionEnd(at("2024-02-29T08:00:00.000Z"), years(6)))).toBe( + "2030-02-28T23:59:59.999Z", + ); + expect(iso(retentionEnd(at("2024-02-29T08:00:00.000Z"), years(4)))).toBe( + "2028-02-29T08:00:00.000Z", + ); + expect(iso(retentionEnd(at("2026-11-30T00:00:00.000Z"), months(3)))).toBe( + "2027-02-28T23:59:59.999Z", + ); + expect(retentionEnd(1_000, { months: 0, ms: 500 })).toBe(1_500); + }); + + it("starts at the latest moment whose period has ended, so a sweep forgets neither early nor late", () => { + expect(iso(retentionStart(at("2032-03-15T10:30:00.000Z"), years(6)))).toBe( + "2026-03-15T10:30:00.000Z", + ); + expect(iso(retentionStart(at("2026-03-31T10:00:00.000Z"), months(1)))).toBe( + "2026-02-28T23:59:59.999Z", + ); + expect(iso(retentionStart(at("2026-02-28T23:59:59.999Z"), months(1)))).toBe( + "2026-01-31T23:59:59.999Z", + ); + expect(iso(retentionStart(at("2026-02-28T12:00:00.000Z"), months(1)))).toBe( + "2026-01-28T12:00:00.000Z", + ); + expect(retentionStart(1_500, { months: 0, ms: 500 })).toBe(1_000); + const day = 86_400_000; + for ( + let start = at("2024-01-25T06:00:00.000Z"); + start < at("2024-04-05T00:00:00.000Z"); + start += day / 4 + ) { + for (const retention of [months(1), months(13), years(1)]) { + const end = retentionEnd(start, retention); + expect(retentionStart(end, retention)).toBeGreaterThanOrEqual(start); + expect(retentionStart(end - 1, retention)).toBeLessThan(start); + } + } + }); +}); diff --git a/packages/core/src/contracts/duration.ts b/packages/core/src/contracts/duration.ts index b7315bee..db29c78d 100644 --- a/packages/core/src/contracts/duration.ts +++ b/packages/core/src/contracts/duration.ts @@ -70,3 +70,107 @@ export const asDuration: AsDurationFunction = (input) => { parseDuration(input); return input as DurationInput; }; + +/** + * The units a retention takes besides a duration's: months and years of the calendar. + */ +export type CalendarUnit = "mo" | "y"; + +/** + * A retention as users write it: milliseconds, a duration string, or a whole number of calendar + * months or years, `"6mo"`, `"6y"`, which a legal period is usually given in and which days + * cannot say, leap years and months of unequal length being what they are. + */ +export type RetentionInput = DurationInput | `${number}${CalendarUnit}`; + +/** + * A retention as the runtime counts it: calendar months, or milliseconds, one of them `0`. + */ +export interface Retention { + readonly months: number; + readonly ms: number; +} + +const CALENDAR_PATTERN = /^(\d+(?:\.\d+)?)(mo|y)$/; + +export interface ParseRetentionFunction { + (input: LooseDurationInput): Retention; +} + +/** + * Throws `ValidationError` for anything but a duration or a whole number of months or years. + */ +export const parseRetention: ParseRetentionFunction = (input) => { + const match = typeof input === "string" ? CALENDAR_PATTERN.exec(input) : null; + if (match === null) return { months: 0, ms: parseDuration(input) }; + const [, amount, unit] = match; + if (!Number.isInteger(Number(amount))) { + throw new ValidationError(`Invalid retention: "${input}"`, [ + { path: [], message: "Expected a whole number of months or years" }, + ]); + } + return { months: Number(amount) * (unit === "y" ? 12 : 1), ms: 0 }; +}; + +const lastDayOf = (year: number, month: number): number => + new Date(Date.UTC(year, month + 1, 0)).getUTCDate(); + +const endOfMonth = (year: number, month: number): number => + Date.UTC(year, month, lastDayOf(year, month), 23, 59, 59, 999); + +// The same day and time `months` later; a day the month lacks, as the 29th of February six years +// on, becomes the very end of the month's last day, so a period is never cut short and a later +// start never ends earlier. +const addMonths = (time: number, months: number): number => { + const at = new Date(time); + const year = at.getUTCFullYear(); + const month = at.getUTCMonth() + months; + const day = at.getUTCDate(); + if (day > lastDayOf(year, month)) return endOfMonth(year, month); + return Date.UTC( + year, + month, + day, + at.getUTCHours(), + at.getUTCMinutes(), + at.getUTCSeconds(), + at.getUTCMilliseconds(), + ); +}; + +export interface RetentionEndFunction { + (start: number, retention: Retention): number; +} + +/** + * When a retention that starts at `start` ends, in milliseconds. + */ +export const retentionEnd: RetentionEndFunction = (start, { months, ms }) => + addMonths(start, months) + ms; + +export interface RetentionStartFunction { + (end: number, retention: Retention): number; +} + +/** + * The latest start whose retention has ended by `end`: every start at or before it has ended, + * every later one has not. + */ +export const retentionStart: RetentionStartFunction = (end, { months, ms }) => { + const by = end - ms; + const at = new Date(by); + const year = at.getUTCFullYear(); + const month = at.getUTCMonth() - months; + // Starts on days the end's month lacks end at the very end of its last day. + const endsTheMonth = by === endOfMonth(year, at.getUTCMonth()); + if (endsTheMonth || at.getUTCDate() > lastDayOf(year, month)) return endOfMonth(year, month); + return Date.UTC( + year, + month, + at.getUTCDate(), + at.getUTCHours(), + at.getUTCMinutes(), + at.getUTCSeconds(), + at.getUTCMilliseconds(), + ); +}; diff --git a/packages/core/src/contracts/index.ts b/packages/core/src/contracts/index.ts index b6028c3c..3588c74b 100644 --- a/packages/core/src/contracts/index.ts +++ b/packages/core/src/contracts/index.ts @@ -15,11 +15,14 @@ export type { export { MAX_IDEMPOTENCY_KEY_LENGTH } from "./command.ts"; export type { AsDurationFunction, + CalendarUnit, DurationInput, DurationString, DurationUnit, LooseDurationInput, ParseDurationFunction, + Retention, + RetentionInput, } from "./duration.ts"; export { asDuration, parseDuration } from "./duration.ts"; export { diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 7a589994..4af3843b 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -50,6 +50,8 @@ export type { GlobalStream, ReadEventsArgs } from "./kernel/events/global-stream export type { MigrateTablesArgs, MigrateTablesFunction } from "./kernel/migrate.ts"; export { migrateTables } from "./kernel/migrate.ts"; export type { + ExportedCategories, + ExportedCategory, ExportedData, ExportedDeadLetter, ExportedEvent, @@ -63,7 +65,10 @@ export type { ForgetLedgerReconciliationStatus, } from "./kernel/personal-data/reconciliation.ts"; export { FORGET_LEDGER_GRACE_MS } from "./kernel/personal-data/reconciliation.ts"; -export type { RetentionStatus } from "./kernel/personal-data/retention.ts"; +export type { + CategoryRetentionStatus, + RetentionStatus, +} from "./kernel/personal-data/retention.ts"; export type { KeyRotationStatus } from "./kernel/personal-data/rotation.ts"; export type { ProcessStatus } from "./kernel/process/lifecycle.ts"; export type { diff --git a/packages/core/src/kernel/access-log.test.ts b/packages/core/src/kernel/access-log.test.ts index 484eb007..2c937369 100644 --- a/packages/core/src/kernel/access-log.test.ts +++ b/packages/core/src/kernel/access-log.test.ts @@ -199,7 +199,9 @@ describe("the access log", () => { const exported = await app.as("operator-1").exportPersonalData("ana"); expect(exported.readModels).toEqual({ - customerDirectory: [{ key: "ana", data: { email: "ana@example.com" } }], + customerDirectory: [ + { key: "ana", data: { email: "ana@example.com" }, categories: { email: "default" } }, + ], customerPlans: [], }); expect(accesses()).toEqual([ diff --git a/packages/core/src/kernel/aggregate/load-aggregate.test.ts b/packages/core/src/kernel/aggregate/load-aggregate.test.ts index f06e9f0f..1487d334 100644 --- a/packages/core/src/kernel/aggregate/load-aggregate.test.ts +++ b/packages/core/src/kernel/aggregate/load-aggregate.test.ts @@ -1,5 +1,6 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import type { EventStore } from "../../adapter/storage/event-store.ts"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import type { Snapshot } from "../../adapter/storage/snapshot-store.ts"; import { testKeyedPerson as held, @@ -180,7 +181,7 @@ describe("createAggregateLoader", () => { it("names whose personal data the state holds, with the key its load opened it with", async () => { const { storage, snapshot, append, load } = await setUp(); - await storage.personalKeys.add("c-0", testPersonalKey("0")); + await storage.personalKeys.add("c-0", DEFAULT_PERSONAL_CATEGORY, testPersonalKey("0")); await storage.snapshotStore.put(snapshot({ persons: [held("c-0", "0")] })); await append( [3, "OrderSigned", { customerId: "c-1", signature: "Ana" }], @@ -190,7 +191,7 @@ describe("createAggregateLoader", () => { ...storage.eventStore, load: async (args) => ({ ...(await storage.eventStore.load(args)), - fingerprints: new Map([["c-1", testPersonalKey("1").fingerprint]]), + opened: [held("c-1", "1")], }), }; expect((await load(opening)).persons).toEqual([ diff --git a/packages/core/src/kernel/aggregate/load-aggregate.ts b/packages/core/src/kernel/aggregate/load-aggregate.ts index e8f964e8..085522b4 100644 --- a/packages/core/src/kernel/aggregate/load-aggregate.ts +++ b/packages/core/src/kernel/aggregate/load-aggregate.ts @@ -131,7 +131,7 @@ export const createAggregateLoader: CreateAggregateLoaderFunction = ({ return { folded: foldState({ aggregate, events }), version: Math.min(loaded.version, through), - persons: personsOf({ aggregate, events, fingerprints: loaded.fingerprints }), + persons: personsOf({ aggregate, events, opened: loaded.opened }), snapshot: null, foldedEvents: events.length, }; @@ -195,7 +195,7 @@ export const createAggregateLoader: CreateAggregateLoaderFunction = ({ const persons = personsOf({ aggregate, events: loaded.events, - fingerprints: loaded.fingerprints, + opened: loaded.opened, from: snapshot.persons, }); const used: LoadedAggregate = { diff --git a/packages/core/src/kernel/app.ts b/packages/core/src/kernel/app.ts index 45630a10..58bf5088 100644 --- a/packages/core/src/kernel/app.ts +++ b/packages/core/src/kernel/app.ts @@ -15,6 +15,7 @@ import { type IdGenerator, uuidV7IdGenerator } from "../contracts/ids.ts"; import { type Logger, silentLogger } from "../contracts/logger.ts"; import { checkActor } from "../contracts/metadata.ts"; import type { OnPersonalDataAccess } from "../modules/audit.ts"; +import type { ForgetOptions } from "../modules/personal-data.ts"; import type { CommandsFacade, QueriesFacade, Registry } from "../modules/registry.ts"; import { validateRegistry } from "../modules/validate.ts"; import type { AppEnv, AppRegistry, EnvSection } from "../register/index.ts"; @@ -37,6 +38,7 @@ import { appStorage } from "./events/app-event-store.ts"; import { createGlobalStream, type GlobalStream } from "./events/global-stream.ts"; import { createAccessLog, recorded } from "./personal-data/access.ts"; import { createPersonalDataBackfill } from "./personal-data/backfill.ts"; +import { buildPersonalCategories, checkMarkedCategories } from "./personal-data/categories.ts"; import { exportPersonalData, type PersonalDataExport } from "./personal-data/export.ts"; import { forgetPersonalData, logForgotten } from "./personal-data/forget.ts"; import { @@ -59,6 +61,7 @@ import { liftProcessingRestriction, restrictProcessing } from "./personal-data/r import { createRetentionSubscriber, createRetentionSweep, + retains, retentionDueOf, } from "./personal-data/retention.ts"; import { createKeyRotation } from "./personal-data/rotation.ts"; @@ -137,18 +140,21 @@ export interface BoundaClient { */ rebuildReadModel(name: string): Promise; /** - * Forgets a person: deletes the key their personal fields are encrypted with, so every event - * of theirs, and the state of every process instance that names them, reads those fields as - * `null` from now on, the instance carrying on, and appends `PersonalDataForgotten` to - * the stream `bounda:`, which every read model that projects personal fields handles. - * Both happen together or not at all. Resolves like a command, so + * Forgets a person: deletes the keys their personal fields are encrypted with, one per + * category, so every event of theirs, and the state of every process instance that names them, + * reads those fields as `null` from now on, the instance carrying on, and appends + * `PersonalDataForgotten` to the stream `bounda:`, which every read model that + * projects personal fields handles. With `categories`, only those, as when a consent is + * withdrawn. A category whose basis is `legal-obligation` is kept instead, restricted, until its + * retention ends. Both happen together or not at all. Resolves like a command, so * `catchUpReadModels({ through })` waits for those read models. Rejects with - * `ValidationError` an empty id. With `runtime.personalData.forgetLedger`, the forgetting is in - * the ledger before it commits: one that rejects after that still forgets the person at the next - * reconciliation, and calling again is safe; a strict ledger that does not take it rejects with - * `ForgetLedgerUnavailableError`, forgetting nobody. + * `ValidationError` an empty id, or categories that are empty or that the app does not have. + * With `runtime.personalData.forgetLedger`, the forgetting is in the ledger before it commits: + * one that rejects after that still forgets what it asked for at the next reconciliation, and + * calling again is safe; a strict ledger that does not take it rejects with + * `ForgetLedgerUnavailableError`, forgetting nothing. */ - forget(personId: string): Promise; + forget(personId: string, options?: ForgetOptions): Promise; /** * Everything the store holds that names a person, for them to take away: their events, with * the fields marked `personal` opened (`null` each once they are forgotten) and nothing else of @@ -265,9 +271,10 @@ export interface BoundaApp extends BoundaClien */ readonly deadLetters: DeadLetters; /** - * Commits again each forgetting an entry names, as a reconciliation does: unless the store holds - * the `PersonalDataForgotten` the entry names by `eventId`, and only while the person holds a key - * it names, any key when it names none; a whole entry under its own id, time and reason, a bare + * Commits again each forgetting an entry names, of the categories it asked for, as a + * reconciliation does: unless the store holds the `PersonalDataForgotten` the entry names by + * `eventId`, and only while the person holds a key in force it covers and names, any such key + * when it names none; a whole entry under its own id, time and reason, a bare * person's id as a request. What `bounda forget --from` runs, for a backup restored without * `runtime.personalData.forgetLedger`; with one, each is recorded in it first, as `forget` does. * Resolves to the ids of the persons it forgot, in order. Rejects with `ValidationError` an entry @@ -547,6 +554,16 @@ export const assembleApp: AssembleAppFunction = async ({ }); const readModels = await buildReadModels({ registry, config, logger }); opened.unshift(() => readModels.close()); + const personalCategories = buildPersonalCategories({ + module: registry.personalData, + retention: config.runtime.personalData.retention, + }); + checkMarkedCategories({ + categories: personalCategories, + aggregates, + processes: processDefinitions, + readModels: readModels.byName, + }); // One list per `runUntilIdle` under way. const observers = new Set(); const pipeline = createCommandPipeline({ @@ -581,6 +598,7 @@ export const assembleApp: AssembleAppFunction = async ({ pipeline, storage, config, + personalCategories, ids, clock, pendingRetries, @@ -592,6 +610,7 @@ export const assembleApp: AssembleAppFunction = async ({ aggregates, pipeline, config, + personalCategories, ids, clock, logger, @@ -630,21 +649,23 @@ export const assembleApp: AssembleAppFunction = async ({ checkpointStore: storage.checkpointStore, following: following.map((subscriber) => subscriber.name), }); - const { retentionMs } = config.runtime.personalData; + // Without a retention the sweep still ends what a forgetting kept that no legal obligation keeps + // any more, which only an app that marks personal fields can hold. + const recordsActivity = retains(personalCategories); const retention = - retentionMs === null - ? undefined - : createRetentionSweep({ + recordsActivity || marksPersonalFields({ aggregates, processes: processDefinitions }) + ? createRetentionSweep({ storage: written, intentions, - retentionMs, + categories: personalCategories, ids, clock, logger, concurrencyRetries: config.runtime.commands.concurrencyRetries, batchSize: config.runtime.dispatcher.batchSize, forgotten: wrapping.drop, - }); + }) + : undefined; const letterRetentionMs = config.runtime.deadLetters.retentionMs; const letterRetention = letterRetentionMs === null @@ -657,14 +678,13 @@ export const assembleApp: AssembleAppFunction = async ({ logger, }); const { concurrencyRetries } = config.runtime.commands; - const retentionDue = - retentionMs === null - ? NEVER_DUE - : retentionDueOf({ - retentionMs, - batchSize: config.runtime.dispatcher.batchSize, - checkpoints: written.checkpointStore, - }); + const retentionDue = recordsActivity + ? retentionDueOf({ + categories: personalCategories, + batchSize: config.runtime.dispatcher.batchSize, + checkpoints: written.checkpointStore, + }) + : NEVER_DUE; const reconciling = forgetLedger === null ? null @@ -679,6 +699,8 @@ export const assembleApp: AssembleAppFunction = async ({ clock, logger, concurrencyRetries, + categories: personalCategories, + batchSize: config.runtime.dispatcher.batchSize, reconcileEveryMs: forgetLedger.config.reconcileEveryMs, keepForMs, forgotten: wrapping.drop, @@ -707,7 +729,7 @@ export const assembleApp: AssembleAppFunction = async ({ wrapping, marks, batchSize: config.runtime.dispatcher.batchSize, - recordsActivity: retention !== undefined, + recordsActivity, clock, logger, errorText, @@ -743,9 +765,9 @@ export const assembleApp: AssembleAppFunction = async ({ ...(test === undefined ? {} : { strict: { onError: test.onSnapshotError } }), }), ), - ...(retention === undefined - ? [] - : [createRetentionSubscriber({ persons: storage.persons })]), + ...(recordsActivity + ? [createRetentionSubscriber({ persons: storage.persons, keys: storage.personalKeys })] + : []), ...(forgetLedger === null ? [] : [createLedgerOutbox({ ledger: forgetLedger.store, keepForMs, clock })]), @@ -944,6 +966,7 @@ export const assembleApp: AssembleAppFunction = async ({ readModels, clock, batchSize: config.runtime.dispatcher.batchSize, + categories: personalCategories, personId, }); return exported; @@ -983,7 +1006,7 @@ export const assembleApp: AssembleAppFunction = async ({ actor, run: () => rebuild(name, actor === undefined ? {} : { actor }), }), - forget: (personId) => + forget: (personId, options) => tracedOperation({ errorText, operation: "forget", @@ -996,7 +1019,10 @@ export const assembleApp: AssembleAppFunction = async ({ clock, logger, concurrencyRetries, + categories: personalCategories, + batchSize: config.runtime.dispatcher.batchSize, personId, + options, actor, }), }), @@ -1118,6 +1144,8 @@ export const assembleApp: AssembleAppFunction = async ({ ids, clock, concurrencyRetries, + categories: personalCategories, + batchSize: config.runtime.dispatcher.batchSize, entry, retentionDue, record: true, diff --git a/packages/core/src/kernel/command/reaction-commands.test.ts b/packages/core/src/kernel/command/reaction-commands.test.ts index a838b2b2..c9587879 100644 --- a/packages/core/src/kernel/command/reaction-commands.test.ts +++ b/packages/core/src/kernel/command/reaction-commands.test.ts @@ -5,6 +5,7 @@ import { pendingEvent } from "../../adapter/testing/fixtures.ts"; import { ValidationError } from "../../contracts/errors.ts"; import { createSequentialIdGenerator } from "../../contracts/ids.ts"; import type { Registry } from "../../modules/registry.ts"; +import { buildPersonalCategories } from "../personal-data/categories.ts"; import { boundForgetLedger } from "../personal-data/forget-ledger.ts"; import { createForgetIntentions } from "../personal-data/intention.ts"; import { createReactionCommandIds } from "../shared/idempotency-key.ts"; @@ -26,6 +27,8 @@ import { const context = { correlationId: "req-1", causationId: "evt-1", depth: 4 }; +const personalCategories = buildPersonalCategories({ module: undefined, retention: null }); + const setUp = async (registry: Registry = orderRegistry) => { const harness = await createKernelHarness({ registry }); const unit = createUnitOfWork({ storage: harness.storage }); @@ -42,6 +45,8 @@ const setUp = async (registry: Registry = orderRegistry) => { clock: harness.clock, intentions: withoutLedger({ ids, clock: harness.clock }), maxChainDepth: 5, + personalCategories, + batchSize: 10, }); return { ...harness, reaction, unit, entries }; }; @@ -67,6 +72,8 @@ const withLedger = async () => { logger, ids, clock: harness.clock, + personalCategories, + batchSize: 10, intentions: createForgetIntentions({ ledger: { store: boundForgetLedger({ @@ -75,7 +82,7 @@ const withLedger = async () => { raiseMark: async () => undefined, append: async (appended) => { recorded.push(...appended); - keysAtRecord.push(await harness.storage.personalKeys.get("c-1")); + keysAtRecord.push(await harness.storage.personalKeys.get("c-1", "default")); }, read: async () => [], }, @@ -374,14 +381,14 @@ describe("createReactionCommands", () => { it("forgets a person when the unit commits, once however often the run names them", async () => { const { reaction, storage, unit, entries } = await setUp(); - await storage.personalKeys.add("c-1", { key: "k", fingerprint: "f", kid: null }); + await storage.personalKeys.add("c-1", "default", { key: "k", fingerprint: "f", kid: null }); await reaction.forget("c-1"); await reaction.forget("c-1"); - expect(await storage.personalKeys.get("c-1")).not.toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).not.toBeNull(); expect(await forgottenEvents(storage)).toEqual([]); await unit.commit(); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).toBeNull(); const [forgotten, ...more] = await forgottenEvents(storage); expect(more).toEqual([]); expect(forgotten).toMatchObject({ @@ -391,6 +398,10 @@ describe("createReactionCommands", () => { payload: { personId: "c-1", reason: "requested", + whole: true, + categories: ["default"], + retained: [], + expired: [], cancelledCommands: [], removedDeadLetters: [], }, @@ -402,7 +413,11 @@ describe("createReactionCommands", () => { fields: { personId: "c-1", reason: "requested", - hadKey: true, + whole: true, + categories: ["default"], + retained: [], + expired: [], + destroyedKeys: 1, cancelledCommands: 0, removedDeadLetters: 0, }, @@ -411,7 +426,11 @@ describe("createReactionCommands", () => { it("records the forgetting's intention in the ledger just before the commit's transaction", async () => { const { harness, unit, reaction, recorded, keysAtRecord } = await withLedger(); - await harness.storage.personalKeys.add("c-1", { key: "k", fingerprint: "f-1", kid: null }); + await harness.storage.personalKeys.add("c-1", "default", { + key: "k", + fingerprint: "f-1", + kid: null, + }); await reaction.forget("c-1"); expect(recorded).toEqual([]); await unit.commit(); @@ -420,6 +439,7 @@ describe("createReactionCommands", () => { expect(recorded).toEqual([ { personId: "c-1", + categories: null, eventId: forgotten?.id, forgottenAt: forgotten?.timestamp, reason: "requested", @@ -432,11 +452,15 @@ describe("createReactionCommands", () => { const { harness, unit, reaction, recorded } = await withLedger(); await reaction.forget("c-1"); unit.beforeCommit(async () => { - await harness.storage.personalKeys.add("c-1", { key: "k", fingerprint: "f-2", kid: null }); + await harness.storage.personalKeys.add("c-1", "default", { + key: "k", + fingerprint: "f-2", + kid: null, + }); }); await unit.commit(); expect(recorded.map(({ fingerprints }) => fingerprints)).toEqual([[]]); - expect(await harness.storage.personalKeys.get("c-1")).toBeNull(); + expect(await harness.storage.personalKeys.get("c-1", "default")).toBeNull(); const [forgotten] = await forgottenEvents(harness.storage); expect(forgotten).toMatchObject({ id: recorded[0]?.eventId, @@ -471,12 +495,31 @@ describe("createReactionCommands", () => { it("forgets nobody for a unit that never commits", async () => { const { reaction, storage } = await setUp(); - await storage.personalKeys.add("c-1", { key: "k", fingerprint: "f", kid: null }); + await storage.personalKeys.add("c-1", "default", { key: "k", fingerprint: "f", kid: null }); await reaction.forget("c-1"); - expect(await storage.personalKeys.get("c-1")).not.toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).not.toBeNull(); expect(await forgottenEvents(storage)).toEqual([]); }); + it("forgets the categories asked for apart from the person as a whole, once each", async () => { + const { reaction, storage, unit } = await setUp(); + await reaction.forget("c-1", { categories: ["default"] }); + await reaction.forget("c-1", { categories: ["default"] }); + await reaction.forget("c-1"); + await unit.commit(); + expect((await forgottenEvents(storage)).map(({ payload }) => payload)).toEqual([ + expect.objectContaining({ whole: false, categories: ["default"] }), + expect.objectContaining({ whole: true, categories: ["default"] }), + ]); + }); + + it("refuses a category the app does not have before the unit commits", async () => { + const { reaction } = await setUp(); + await expect( + reaction.forget("c-1", { categories: ["profile"] as unknown as ["default"] }), + ).rejects.toThrow('forget names categories the app does not have: "profile"'); + }); + it("refuses to forget an empty id, and once the run is abandoned or has finished", async () => { const { reaction } = await setUp(); await expect(reaction.forget("")).rejects.toThrow( @@ -504,6 +547,8 @@ describe("createReactionCommands", () => { clock: harness.clock, intentions: withoutLedger({ ids: deepIds, clock: harness.clock }), maxChainDepth: 5, + personalCategories, + batchSize: 10, }); await expect(deep.forget("c-1")).rejects.toMatchObject({ code: "CHAIN_DEPTH_EXCEEDED", diff --git a/packages/core/src/kernel/command/reaction-commands.ts b/packages/core/src/kernel/command/reaction-commands.ts index 836d4a3b..a07b21a5 100644 --- a/packages/core/src/kernel/command/reaction-commands.ts +++ b/packages/core/src/kernel/command/reaction-commands.ts @@ -8,8 +8,15 @@ import { BoundaError, ChainDepthExceededError } from "../../contracts/errors.ts" import type { IdGenerator } from "../../contracts/ids.ts"; import type { Logger } from "../../contracts/logger.ts"; import type { CausationContext } from "../../contracts/metadata.ts"; +import type { ForgetOptions } from "../../modules/personal-data.ts"; import type { AggregatesRuntime } from "../aggregate/runtime.ts"; -import { forgetWithin, logForgotten, STALE_INTENTION } from "../personal-data/forget.ts"; +import type { PersonalCategories } from "../personal-data/categories.ts"; +import { + forgetScopeOf, + forgetWithin, + logForgotten, + STALE_INTENTION, +} from "../personal-data/forget.ts"; import type { ForgetIntention, ForgetIntentions } from "../personal-data/intention.ts"; import { checkPersonId } from "../personal-data/transact.ts"; import { createReactionCommandIds } from "../shared/idempotency-key.ts"; @@ -63,7 +70,7 @@ export interface ReactionCommands { * before the transaction opens, so an attempt whose transaction then fails leaves it for the * next reconciliation, which commits it. */ - forget(personId: string): Promise; + forget(personId: string, options?: ForgetOptions): Promise; /** * What the run's commands decided, in the order they finished, once every dispatch, awaited by * the handler or not, has settled; rejects with the first that failed. The run awaits it before @@ -102,6 +109,11 @@ export interface CreateReactionCommandsArgs { * How deep a `forget` may go, which no pipeline checks for it as it does for a command. */ readonly maxChainDepth: number; + readonly personalCategories: PersonalCategories; + /** + * How many events a `forget` reads at once. + */ + readonly batchSize: number; } export interface CreateReactionCommandsFunction { @@ -126,6 +138,8 @@ export const createReactionCommands: CreateReactionCommandsFunction = ({ clock, intentions, maxChainDepth, + personalCategories, + batchSize, }) => { const commandIds = createReactionCommandIds(idempotencyKey); const forgetting = new Set(); @@ -203,8 +217,9 @@ export const createReactionCommands: CreateReactionCommandsFunction = ({ return { commands, signal: handler.signal, - forget: async (personId) => { + forget: async (personId, options) => { checkPersonId(personId, "forget"); + const scope = forgetScopeOf({ categories: personalCategories, options }); if (abandoned.signal.aborted) throw abandoned.signal.reason; if (finished) { throw new ReactionFinishedError( @@ -214,13 +229,15 @@ export const createReactionCommands: CreateReactionCommandsFunction = ({ } const depth = context.depth + 1; if (depth > maxChainDepth) throw new ChainDepthExceededError(depth, maxChainDepth); - if (forgetting.has(personId)) return; - forgetting.add(personId); + const asked = JSON.stringify([personId, scope]); + if (forgetting.has(asked)) return; + forgetting.add(asked); let intention: ForgetIntention | undefined; within.beforeCommit(async (storage) => { intention = await intentions.intend({ personId, reason: "requested", + scope, keys: storage.personalKeys, }); }); @@ -234,6 +251,9 @@ export const createReactionCommands: CreateReactionCommandsFunction = ({ personId, intention: named, context: { ...context, depth }, + categories: personalCategories, + scope, + batchSize, }); // A key given after the intention goes too: the commit is one, and a run whose commit // fails is delivered again, with an intention that names it. diff --git a/packages/core/src/kernel/dispatch/subscriber-failures-app.test.ts b/packages/core/src/kernel/dispatch/subscriber-failures-app.test.ts index 93476859..69bbcac9 100644 --- a/packages/core/src/kernel/dispatch/subscriber-failures-app.test.ts +++ b/packages/core/src/kernel/dispatch/subscriber-failures-app.test.ts @@ -124,6 +124,42 @@ const registry = { }, } as unknown as Registry; +// The same app, with a marketing category its customers subscribe under. +const customer = (registry.aggregates as Record).customer; +const categorized = { + ...registry, + aggregates: { + customer: { + ...customer, + events: { + ...customer?.events, + customerSubscribed: { + payload: ({ z, personal }: EventPayloadArgs) => + z.object({ topic: personal(z.string(), { category: "marketing" }) }), + evolve: () => ({}), + }, + }, + commands: { + ...customer?.commands, + subscribeCustomer: { + module: { + payload: ({ z }: PayloadArgs) => + z.object({ customerId: z.string(), topic: z.string() }), + handler: ({ + command, + events, + }: { + command: { payload: { topic: string } }; + events: Record unknown>; + }) => [events.customerSubscribed?.({ topic: command.payload.topic })], + }, + }, + }, + }, + }, + personalData: { categories: { marketing: { basis: "consent", retention: "2y" } } }, +} as unknown as Registry; + const SUBSCRIBER = "projection:customers"; interface Logged { @@ -149,11 +185,11 @@ afterEach(async () => { let started = 0; -const start = async (adapter: Adapter) => { +const start = async (adapter: Adapter, of: Registry = registry) => { started += 1; const logger = recordingLogger(); const app = await createApp({ - registry, + registry: of, config: { storage: adapter }, ids: createSequentialIdGenerator({ prefix: `app-${started}` }), clock: createFixedClock(new Date("2026-05-01T00:00:00.000Z")), @@ -265,6 +301,29 @@ describe.each([ }); }); + it("reads the text as null once any category of the person is forgotten, since it may quote any", async () => { + const adapter = adapterOf(); + const { app } = await start(adapter, categorized); + await app.commands.subscribeCustomer?.({ customerId: "c-1", topic: "shoes" }); + failing.on = true; + await register(app, "c-1"); + await app.runUntilIdle({ maxPasses: 3 }); + expect(await app.subscriberFailures.get(SUBSCRIBER)).toMatchObject({ + errorMessage: "cannot project c-1@example.com", + }); + await app.forget("c-1", { categories: ["marketing"] }); + expect(await app.subscriberFailures.get(SUBSCRIBER)).toMatchObject({ + person: "c-1", + errorMessage: null, + errorStack: null, + }); + expect((await app.exportPersonalData("c-1")).events.map(({ data }) => data)).toEqual([ + { topic: null }, + { email: "c-1@example.com" }, + {}, + ]); + }); + it("keeps in the clear the text of a failure on an event that names nobody", async () => { const adapter = adapterOf(); const { app } = await start(adapter); @@ -298,7 +357,7 @@ describe("the status of the app", () => { const { adapter } = createNodeSqliteAdapter(); const { app } = await start(adapter); expect((await app.status()).schema).toEqual([ - { database: "storage", scope: "storage", version: 2, latest: 2 }, + { database: "storage", scope: "storage", version: 3, latest: 3 }, { database: "storage", scope: "checkpoints", version: 1, latest: 1 }, ]); const inMemory = await start(memory()); @@ -309,7 +368,7 @@ describe("the status of the app", () => { personalDataBackfill: { caughtUp: false }, forgetLedger: null, keyRotation: null, - retention: null, + retention: { categories: [], keptBeyond: 0 }, deadLetterRetention: null, }); }); @@ -337,14 +396,17 @@ describe("the status of the app", () => { await app.runUntilIdle(); const status = await app.status(); expect(status.schema).toEqual([ - { database: "storage", scope: "storage", version: 2, latest: 2 }, + { database: "storage", scope: "storage", version: 3, latest: 3 }, { database: "storage", scope: "checkpoints", version: 1, latest: 1 }, { database: "readModels.customers", scope: "checkpoints", version: 1, latest: 1 }, - { database: "forgetLedger", scope: "forget-ledger", version: 3, latest: 3 }, + { database: "forgetLedger", scope: "forget-ledger", version: 4, latest: 4 }, ]); expect(status).toMatchObject({ forgetLedger: { reconciled: true, held: null }, - retention: { due: 0, nextDueAt: null }, + retention: { + categories: [{ category: "default", retention: "30d", due: 0, nextDueAt: null }], + keptBeyond: 0, + }, deadLetterRetention: { nextDueAt: null }, }); }); diff --git a/packages/core/src/kernel/events/app-event-store.test.ts b/packages/core/src/kernel/events/app-event-store.test.ts index 13379f38..a8dca123 100644 --- a/packages/core/src/kernel/events/app-event-store.test.ts +++ b/packages/core/src/kernel/events/app-event-store.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from "vitest"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { createFixedClock } from "../../contracts/clock.ts"; import { memory } from "../../memory/index.ts"; import { buildAggregates } from "../aggregate/build-aggregates.ts"; @@ -27,8 +28,8 @@ const setUp = async () => { processes: { byName: {} }, wrapping, }); - await storage.personalKeys.create("c-1"); - const stored = await written.personalKeys.get("c-1"); + await storage.personalKeys.create("c-1", DEFAULT_PERSONAL_CATEGORY); + const stored = await written.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); if (stored === null) throw new Error("no key stored"); // A read that started before a removal, as a projection's, keeps the key again. const readMeanwhile = () => wrapping.over(written.personalKeys).open("c-1", stored); @@ -40,26 +41,26 @@ describe("appStorage", () => { const { unwrapped, written, storage, stored, readMeanwhile } = await setUp(); expect(unwrapped).toHaveLength(0); await storage.transact(async (transaction) => { - await transaction.personalKeys.remove("c-1"); + await transaction.personalKeys.remove("c-1", stored.fingerprint); await readMeanwhile(); }); expect(unwrapped).toHaveLength(1); // The row put back, as a restore would: unwrapped again, so the cache had let it go. - await written.personalKeys.add("c-1", stored); - await storage.personalKeys.held("c-1"); + await written.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, stored); + await storage.personalKeys.held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(unwrapped).toHaveLength(2); }); it("keeps a key whose removal did not commit", async () => { - const { unwrapped, storage, readMeanwhile } = await setUp(); + const { unwrapped, storage, stored, readMeanwhile } = await setUp(); await expect( storage.transact(async (transaction) => { - await transaction.personalKeys.remove("c-1"); + await transaction.personalKeys.remove("c-1", stored.fingerprint); await readMeanwhile(); throw new Error("disk full"); }), ).rejects.toThrow("disk full"); - expect(await storage.personalKeys.held("c-1")).not.toBeNull(); + expect(await storage.personalKeys.held("c-1", DEFAULT_PERSONAL_CATEGORY)).not.toBeNull(); expect(unwrapped).toHaveLength(1); }); }); diff --git a/packages/core/src/kernel/events/app-event-store.ts b/packages/core/src/kernel/events/app-event-store.ts index 065d50ac..c8446393 100644 --- a/packages/core/src/kernel/events/app-event-store.ts +++ b/packages/core/src/kernel/events/app-event-store.ts @@ -54,9 +54,9 @@ export const appStorage: AppStorageFunction = ({ storage, aggregates, processes, const over = wrapping.over(transaction.personalKeys); const keys = { ...over, - remove: (person: string) => { + remove: (person: string, fingerprint: string) => { removed.push(person); - return over.remove(person); + return over.remove(person, fingerprint); }, }; return work({ diff --git a/packages/core/src/kernel/migrate.test.ts b/packages/core/src/kernel/migrate.test.ts index ca193782..4710996f 100644 --- a/packages/core/src/kernel/migrate.test.ts +++ b/packages/core/src/kernel/migrate.test.ts @@ -25,10 +25,10 @@ describe("migrateTables", () => { runtime: { personalData: { forgetLedger: { store: ledger.adapter } } }, }; expect(await migrateTables({ config })).toEqual([ - { database: "storage", scope: "storage", version: 2, latest: 2 }, + { database: "storage", scope: "storage", version: 3, latest: 3 }, { database: "storage", scope: "checkpoints", version: 1, latest: 1 }, { database: "readModels.customers", scope: "checkpoints", version: 1, latest: 1 }, - { database: "forgetLedger", scope: "forget-ledger", version: 3, latest: 3 }, + { database: "forgetLedger", scope: "forget-ledger", version: 4, latest: 4 }, ]); expect(tablesIn(readModels.db)).toEqual(["bounda_checkpoints", "bounda_migrations"]); expect(tablesIn(ledger.db)).toEqual([ diff --git a/packages/core/src/kernel/personal-data/activity.test.ts b/packages/core/src/kernel/personal-data/activity.test.ts new file mode 100644 index 00000000..b250ac25 --- /dev/null +++ b/packages/core/src/kernel/personal-data/activity.test.ts @@ -0,0 +1,114 @@ +import { describe, expect, it } from "vitest"; +import type { StoredEvent } from "../../contracts/event.ts"; +import { memory } from "../../memory/index.ts"; +import { isActivity, lastActivity } from "./activity.ts"; + +const event = ( + position: number, + aggregateType: string, + timestamp: string, + person: string | null = "c-1", +): StoredEvent => ({ + id: `e-${position}`, + aggregateType, + aggregateId: "c-1", + version: position, + type: "Happened", + payload: {}, + timestamp, + metadata: { correlationId: "c", causationId: "c", depth: 0, schemaVersion: 1, system: false }, + person, + position, +}); + +describe("isActivity", () => { + it("counts the events of the app's aggregates, never the runtime's own", () => { + expect(isActivity(event(1, "customer", "2026-01-01T00:00:00.000Z"))).toBe(true); + expect(isActivity(event(1, "bounda", "2026-01-01T00:00:00.000Z"))).toBe(false); + }); +}); + +describe("lastActivity", () => { + const at = (day: number): string => new Date(Date.UTC(2026, 0, day)).toISOString(); + + const stores = ( + events: readonly StoredEvent[], + row: { lastActiveAt: string | null; position: number } | null, + ) => { + const reads: number[] = []; + return { + reads, + persons: { + get: async () => + row === null + ? null + : { + person: "c-1", + restrictedSince: null, + forgottenAt: null, + hold: null, + ...row, + }, + }, + eventStore: { + readByPerson: async ({ + afterPosition, + limit, + }: { + afterPosition: number; + limit: number; + }) => { + reads.push(afterPosition); + return events.filter((e) => e.position > afterPosition).slice(0, limit); + }, + }, + }; + }; + + it("takes the row's last activity when nothing past it names the person", async () => { + const { persons, eventStore, reads } = stores([], { lastActiveAt: at(3), position: 7 }); + expect(await lastActivity({ persons, eventStore, person: "c-1", batchSize: 2 })).toBe(at(3)); + expect(reads).toEqual([7]); + }); + + it("reads past the row, page by page, the activity it has not recorded yet, and skips the runtime's own events", async () => { + const events = [ + event(8, "customer", at(4)), + event(9, "bounda", at(9)), + event(10, "order", at(6)), + event(11, "order", at(5)), + event(12, "bounda", at(10)), + ]; + const { persons, eventStore, reads } = stores(events, { lastActiveAt: at(3), position: 7 }); + expect(await lastActivity({ persons, eventStore, person: "c-1", batchSize: 2 })).toBe(at(6)); + expect(reads).toEqual([7, 9, 11]); + }); + + it("keeps the row's activity over older events it has not recorded", async () => { + const { persons, eventStore } = stores([event(8, "customer", at(1))], { + lastActiveAt: at(3), + position: 7, + }); + expect(await lastActivity({ persons, eventStore, person: "c-1", batchSize: 5 })).toBe(at(3)); + }); + + it("reads from the start for a person without a row, and answers null for one never active", async () => { + const active = stores([event(1, "customer", at(2))], null); + expect(await lastActivity({ ...active, person: "c-1", batchSize: 5 })).toBe(at(2)); + expect(active.reads).toEqual([0]); + const idle = stores([event(1, "bounda", at(2))], null); + expect(await lastActivity({ ...idle, person: "c-1", batchSize: 5 })).toBeNull(); + }); + + it("reads a real store by person", async () => { + const storage = await memory().createStorage(); + expect( + await lastActivity({ + persons: storage.persons, + eventStore: storage.eventStore, + person: "c-1", + batchSize: 5, + }), + ).toBeNull(); + }); +}); diff --git a/packages/core/src/kernel/personal-data/activity.ts b/packages/core/src/kernel/personal-data/activity.ts new file mode 100644 index 00000000..94145072 --- /dev/null +++ b/packages/core/src/kernel/personal-data/activity.ts @@ -0,0 +1,53 @@ +import type { EventStore } from "../../adapter/storage/event-store.ts"; +import type { PersonStore } from "../../adapter/storage/person-store.ts"; +import type { StoredEvent } from "../../contracts/event.ts"; +import { BOUNDA_AGGREGATE } from "../../modules/bounda-events.ts"; + +export interface IsActivityFunction { + (event: StoredEvent): boolean; +} + +/** + * Whether an event is activity of the person it names: an event of the app's aggregates. The + * runtime's own events on `bounda:` are not. + */ +export const isActivity: IsActivityFunction = (event) => event.aggregateType !== BOUNDA_AGGREGATE; + +export interface LastActivityArgs { + readonly persons: Pick; + readonly eventStore: Pick; + readonly person: string; + readonly batchSize: number; +} + +export interface LastActivityFunction { + (args: LastActivityArgs): Promise; +} + +/** + * The timestamp of the person's last activity: what their row records, and past it what the + * retention's subscriber has not recorded yet, read from the stream. `null` when they were never + * active. + */ +export const lastActivity: LastActivityFunction = async ({ + persons, + eventStore, + person, + batchSize, +}) => { + const row = await persons.get(person); + let last = row?.lastActiveAt ?? null; + for (let after = row?.position ?? 0; ; ) { + const events = await eventStore.readByPerson({ + person, + afterPosition: after, + limit: batchSize, + }); + for (const event of events) { + if (isActivity(event) && (last === null || event.timestamp > last)) last = event.timestamp; + } + const tail = events[batchSize - 1]; + if (tail === undefined) return last; + after = tail.position; + } +}; diff --git a/packages/core/src/kernel/personal-data/backfill-walk.test.ts b/packages/core/src/kernel/personal-data/backfill-walk.test.ts index 8c642a48..949b7f96 100644 --- a/packages/core/src/kernel/personal-data/backfill-walk.test.ts +++ b/packages/core/src/kernel/personal-data/backfill-walk.test.ts @@ -26,16 +26,22 @@ const aggregates = { customer: { name: "customer", snapshot: null, - eventsByType: { CustomerRegistered: { personal: { person: null, fields: ["email"] } } }, + eventsByType: { + CustomerRegistered: { personal: { person: null, fields: ["email"], categories: {} } }, + }, }, order: { name: "order", snapshot: null, - eventsByType: { OrderPlaced: { personal: { person: "customerId", fields: ["address"] } } }, + eventsByType: { + OrderPlaced: { personal: { person: "customerId", fields: ["address"], categories: {} } }, + }, }, }, commandsByType: { - ShipOrder: { command: { personal: { person: "customerId", fields: ["address"] } } }, + ShipOrder: { + command: { personal: { person: "customerId", fields: ["address"], categories: {} } }, + }, }, } as unknown as AggregatesRuntime; diff --git a/packages/core/src/kernel/personal-data/backfill.test.ts b/packages/core/src/kernel/personal-data/backfill.test.ts index 1e0b2956..89ee9946 100644 --- a/packages/core/src/kernel/personal-data/backfill.test.ts +++ b/packages/core/src/kernel/personal-data/backfill.test.ts @@ -237,7 +237,7 @@ describe.each(adapters)("the backfill of personal data on %s", (_name, adapterOf failing.remind = false; const storage = await adapter.createStorage(); expect(await storage.deadLetterStore.count()).toBe(2); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).toBeNull(); const { logger, logs } = recording(); const after = await start({ @@ -251,7 +251,7 @@ describe.each(adapters)("the backfill of personal data on %s", (_name, adapterOf }); await after.app.runUntilIdle(); - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", "default"); // Without a retention, nobody's activity is recorded. expect(await storage.persons.get("c-1")).toBeNull(); const events = await raw(adapter); @@ -336,8 +336,8 @@ describe.each(adapters)("the backfill of personal data on %s", (_name, adapterOf await after.app.runUntilIdle(); const storage = await adapter.createStorage(); - expect(await storage.personalKeys.get("c-1")).toBeNull(); - const key = await storage.personalKeys.get("c-2"); + expect(await storage.personalKeys.get("c-1", "default")).toBeNull(); + const key = await storage.personalKeys.get("c-2", "default"); const sealedUnder = registered(await raw(adapter)).map( (event) => (event.payload as { email: { $personal: { key: string } } }).email.$personal.key, ); @@ -581,3 +581,38 @@ describe("the backfill of personal data", () => { expect(await app.app.nextDueAt()).toBeNull(); }); }); + +describe("marks walked again once others walked since", () => { + const startedRow = async (adapter: Adapter): Promise<[string, number]> => { + const rows = await (await adapter.createStorage()).checkpointStore.list(); + const row = rows.find( + ({ subscriber }) => subscriber.startsWith(BACKFILL_PREFIX) && subscriber.endsWith(":started"), + ); + if (row === undefined) throw new Error("no walk started"); + return [row.subscriber, row.position]; + }; + + it("walks again only when other marks started later, or at the same moment under a later name", async () => { + const adapter = memory(); + const { app, clock } = await start({ adapter, marks: marked }); + await app.runUntilIdle(); + const [own, first] = await startedRow(adapter); + const digest = own.slice(BACKFILL_PREFIX.length).split(":")[0] ?? ""; + const checkpoints = (await adapter.createStorage()).checkpointStore; + const pass = async (): Promise => { + clock.advance(BACKFILL_FOLLOW_INTERVAL_MS); + await app.runUntilIdle(); + return checkpoints.get(own); + }; + await checkpoints.set(`${BACKFILL_PREFIX}other:position`, first + 10 * MINUTE_MS); + await checkpoints.set(`${BACKFILL_PREFIX}0:started`, first); + await checkpoints.set(`${BACKFILL_PREFIX}older:started`, first - 1); + await checkpoints.set("projection:x", first + 10 * MINUTE_MS); + expect(await pass()).toBe(first); + await checkpoints.set(`${BACKFILL_PREFIX}${digest}z:started`, first); + const again = await pass(); + expect(again).toBeGreaterThan(first); + await checkpoints.set(`${BACKFILL_PREFIX}newer:started`, again + 1); + expect(await pass()).toBeGreaterThan(again); + }); +}); diff --git a/packages/core/src/kernel/personal-data/backfill.ts b/packages/core/src/kernel/personal-data/backfill.ts index 926d2f06..9597adbb 100644 --- a/packages/core/src/kernel/personal-data/backfill.ts +++ b/packages/core/src/kernel/personal-data/backfill.ts @@ -232,6 +232,19 @@ export const createPersonalDataBackfill: CreatePersonalDataBackfillFunction = ({ } }; + // Marks walked since these were: what they sealed follows them, as a field moved to another + // category, so these walk again from the first event, as when a deploy goes back to them. + const walkedSince = async (started: number): Promise => { + for (const { subscriber, position } of await checkpointStore.list()) { + if (!subscriber.startsWith(BACKFILL_PREFIX)) continue; + const [owner = "", name] = subscriber.slice(BACKFILL_PREFIX.length).split(":"); + // Marks started at the same moment are ordered by name, so only one of them walks again. + const later = position > started || (position === started && owner > marks); + if (owner !== marks && name === "started" && later) return true; + } + return false; + }; + const report = (): void => { logger.info("personal data written before its marks is sealed", { marks, ...tally }); tally = { events: 0, discarded: 0, unsealable: 0, scheduledCommands: 0, deadLetters: 0 }; @@ -344,6 +357,13 @@ export const createPersonalDataBackfill: CreatePersonalDataBackfillFunction = ({ } if (!(await lease.take(at))) return 0; let started = await checkpointStore.get(rows.started); + if (started !== 0 && (await walkedSince(started))) { + for (const row of [rows.position, rows.stores, rows.started]) { + await checkpointStore.remove(row); + } + caughtUp = false; + started = 0; + } if (started === 0) { await checkpointStore.compareAndSet(rows.started, 0, at); started = await checkpointStore.get(rows.started); diff --git a/packages/core/src/kernel/personal-data/categories.test.ts b/packages/core/src/kernel/personal-data/categories.test.ts new file mode 100644 index 00000000..b74d2003 --- /dev/null +++ b/packages/core/src/kernel/personal-data/categories.test.ts @@ -0,0 +1,178 @@ +import { describe, expect, it } from "vitest"; +import { ConfigurationError } from "../../contracts/errors.ts"; +import type { PersonalDataModule } from "../../modules/personal-data.ts"; +import { + buildPersonalCategories, + checkCategory, + checkMarkedCategories, + formatRetention, +} from "./categories.ts"; + +const module = (categories: unknown): PersonalDataModule => ({ categories }) as PersonalDataModule; + +const refusal = (categories: unknown): string => { + try { + buildPersonalCategories({ module: module(categories), retention: null }); + } catch (error) { + expect(error).toBeInstanceOf(ConfigurationError); + return (error as Error).message; + } + throw new Error("expected a refusal"); +}; + +describe("buildPersonalCategories", () => { + it("puts the default category first, with the app's retention and no basis, then the declared ones in order", () => { + const categories = buildPersonalCategories({ + module: { + categories: { + profile: { basis: "contract", retention: "30d" }, + invoices: { basis: "legal-obligation", retention: "6y" }, + }, + }, + retention: { months: 0, ms: 5_000 }, + }); + expect(categories.all).toEqual([ + { name: "default", basis: null, retention: { months: 0, ms: 5_000 } }, + { name: "profile", basis: "contract", retention: { months: 0, ms: 2_592_000_000 } }, + { name: "invoices", basis: "legal-obligation", retention: { months: 72, ms: 0 } }, + ]); + expect(categories.declared.map(({ name }) => name)).toEqual(["profile", "invoices"]); + expect(categories.of("invoices")?.basis).toBe("legal-obligation"); + expect(categories.of("default")?.retention).toEqual({ months: 0, ms: 5_000 }); + expect(categories.of("nope")).toBeUndefined(); + }); + + it("has the default category alone without app/personal-data.ts", () => { + const categories = buildPersonalCategories({ module: undefined, retention: null }); + expect(categories.all).toEqual([{ name: "default", basis: null, retention: null }]); + expect(categories.declared).toEqual([]); + }); + + it("refuses what the types would have refused, naming the category and the rule", () => { + expect(refusal(null)).toBe("app/personal-data.ts must export categories as an object"); + expect(refusal(["profile"])).toBe("app/personal-data.ts must export categories as an object"); + expect(refusal("profile")).toBe("app/personal-data.ts must export categories as an object"); + expect(refusal({ default: { basis: "consent", retention: "1d" } })).toBe( + 'app/personal-data.ts: category "default" is the category of fields marked without one, and cannot be declared', + ); + for (const name of ["1st", "with space", "-dash", ""]) { + expect(refusal({ [name]: { basis: "consent", retention: "1d" } })).toBe( + `app/personal-data.ts: category "${name}" must start with a letter and hold only letters, digits, "-" and "_"`, + ); + } + expect(refusal({ a_b: { basis: "whim", retention: "1d" } })).toBe( + 'app/personal-data.ts: category "a_b" must name its basis, one of "consent", "contract", "legal-obligation", "vital-interests", "public-task", "legitimate-interests"', + ); + expect(refusal({ "a-1": null })).toContain("must name its basis"); + expect(refusal({ profile: { basis: "consent", retention: "a month" } })).toBe( + 'app/personal-data.ts: category "profile" must give its retention as a duration, such as "30d", milliseconds, or whole months or years, such as "6y"', + ); + expect(refusal({ profile: { basis: "consent", retention: 0 } })).toBe( + 'app/personal-data.ts: category "profile" must keep its data for more than no time', + ); + expect( + buildPersonalCategories({ + module: module({ p: { basis: "consent", retention: 1 } }), + retention: null, + }).of("p")?.retention, + ).toEqual({ months: 0, ms: 1 }); + }); +}); + +describe("checkMarkedCategories", () => { + const categories = buildPersonalCategories({ + module: { categories: { profile: { basis: "consent", retention: "1d" } } }, + retention: null, + }); + const marked = (fields: Record) => ({ personal: { categories: fields } }); + const check = ( + places: Partial[0], "categories">>, + ): void => + checkMarkedCategories({ + categories, + aggregates: { byName: {} }, + processes: { all: [] }, + readModels: {}, + ...places, + }); + + it("lets through marks in a category the app has, and fields without a mark", () => { + expect(() => + check({ + aggregates: { + byName: { + customer: { + eventsByType: { CustomerRegistered: marked({ email: "profile", note: "default" }) }, + commands: { RegisterCustomer: { personal: null } }, + }, + }, + }, + processes: { all: [{ name: "onboarding", ...marked({ email: "profile" }) }] }, + readModels: { + directory: { fields: { email: { category: "profile" }, id: { category: null } } }, + }, + }), + ).not.toThrow(); + }); + + it("names where a mark is in a category the app does not declare", () => { + const refused = (places: Parameters[0]): string => { + try { + check(places); + } catch (error) { + expect(error).toBeInstanceOf(ConfigurationError); + return (error as Error).message; + } + throw new Error("expected a refusal"); + }; + const unknown = + 'is marked personal in the category "billing", which app/personal-data.ts does not declare'; + expect( + refused({ + aggregates: { + byName: { + order: { eventsByType: { OrderPlaced: marked({ address: "billing" }) }, commands: {} }, + }, + }, + }), + ).toBe(`Event order.OrderPlaced, field "address" ${unknown}`); + expect( + refused({ + aggregates: { + byName: { + order: { eventsByType: {}, commands: { PlaceOrder: marked({ address: "billing" }) } }, + }, + }, + }), + ).toBe(`Command order.PlaceOrder, field "address" ${unknown}`); + expect( + refused({ processes: { all: [{ name: "payment", ...marked({ address: "billing" }) }] } }), + ).toBe(`State of process payment, field "address" ${unknown}`); + expect( + refused({ readModels: { orders: { fields: { address: { category: "billing" } } } } }), + ).toBe(`Read model orders, field "address" ${unknown}`); + }); + + it("checks one category at a time", () => { + expect(() => checkCategory({ categories, category: "profile", at: "x" })).not.toThrow(); + expect(() => checkCategory({ categories, category: "default", at: "x" })).not.toThrow(); + expect(() => checkCategory({ categories, category: "nope", at: "x" })).toThrow( + ConfigurationError, + ); + }); +}); + +describe("formatRetention", () => { + it("writes a retention in the largest unit that divides it", () => { + const ms = (value: number) => formatRetention({ months: 0, ms: value }); + expect(ms(2 * 86_400_000)).toBe("2d"); + expect(ms(3 * 3_600_000)).toBe("3h"); + expect(ms(90 * 60_000)).toBe("90m"); + expect(ms(5_000)).toBe("5s"); + expect(ms(1_500)).toBe("1500ms"); + expect(ms(0)).toBe("0d"); + expect(formatRetention({ months: 72, ms: 0 })).toBe("6y"); + expect(formatRetention({ months: 18, ms: 0 })).toBe("18mo"); + expect(formatRetention({ months: 1, ms: 0 })).toBe("1mo"); + }); +}); diff --git a/packages/core/src/kernel/personal-data/categories.ts b/packages/core/src/kernel/personal-data/categories.ts new file mode 100644 index 00000000..9e7486cc --- /dev/null +++ b/packages/core/src/kernel/personal-data/categories.ts @@ -0,0 +1,202 @@ +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; +import { parseRetention, type Retention } from "../../contracts/duration.ts"; +import { ConfigurationError } from "../../contracts/errors.ts"; +import { + LEGAL_BASES, + type LegalBasis, + type PersonalDataModule, +} from "../../modules/personal-data.ts"; + +/** + * A category as the runtime uses it: its basis, `null` for the default category, which declares + * none, and its retention, `null` when nothing forgets it for inactivity, as the default category + * without `runtime.personalData.retention`. + */ +export interface PersonalCategory { + readonly name: string; + readonly basis: LegalBasis | null; + readonly retention: Retention | null; +} + +/** + * Every category of the app, the default one first, then those `app/personal-data.ts` declares in + * the order it declares them. + */ +export interface PersonalCategories { + readonly all: readonly PersonalCategory[]; + /** + * The declared ones alone, whose retention a sweep of their keys applies. + */ + readonly declared: readonly PersonalCategory[]; + of(name: string): PersonalCategory | undefined; +} + +export interface BuildPersonalCategoriesArgs { + readonly module: PersonalDataModule | undefined; + /** + * `runtime.personalData.retention`, the default category's. + */ + readonly retention: Retention | null; +} + +export interface BuildPersonalCategoriesFunction { + (args: BuildPersonalCategoriesArgs): PersonalCategories; +} + +const NAME = /^[A-Za-z][A-Za-z0-9_-]*$/; + +const declaredIn = (module: PersonalDataModule | undefined): PersonalCategory[] => { + if (module === undefined) return []; + const { categories } = module; + if (typeof categories !== "object" || categories === null || Array.isArray(categories)) { + throw new ConfigurationError("app/personal-data.ts must export categories as an object"); + } + return Object.entries(categories).map(([name, category]) => { + const at = `app/personal-data.ts: category "${name}"`; + if (name === DEFAULT_PERSONAL_CATEGORY) { + throw new ConfigurationError( + `${at} is the category of fields marked without one, and cannot be declared`, + ); + } + if (!NAME.test(name)) { + throw new ConfigurationError( + `${at} must start with a letter and hold only letters, digits, "-" and "_"`, + ); + } + if (!LEGAL_BASES.includes(category?.basis)) { + throw new ConfigurationError( + `${at} must name its basis, one of ${LEGAL_BASES.map((basis) => `"${basis}"`).join(", ")}`, + ); + } + let retention: Retention; + try { + retention = parseRetention(category.retention); + } catch { + throw new ConfigurationError( + `${at} must give its retention as a duration, such as "30d", milliseconds, or whole months or years, such as "6y"`, + ); + } + if (retention.months === 0 && retention.ms <= 0) { + throw new ConfigurationError(`${at} must keep its data for more than no time`); + } + return { name, basis: category.basis, retention }; + }); +}; + +/** + * Throws `ConfigurationError` for a declaration the types would have refused, for a project that + * skipped the type check. + */ +export const buildPersonalCategories: BuildPersonalCategoriesFunction = ({ module, retention }) => { + const declared = declaredIn(module); + const all: PersonalCategory[] = [ + { name: DEFAULT_PERSONAL_CATEGORY, basis: null, retention }, + ...declared, + ]; + const byName = new Map(all.map((category) => [category.name, category])); + return { all, declared, of: (name) => byName.get(name) }; +}; + +export interface CheckCategoryArgs { + readonly categories: PersonalCategories; + readonly category: string; + /** + * Where the mark is, for the message: `event customer.CustomerRegistered, field "email"`. + */ + readonly at: string; +} + +export interface CheckCategoryFunction { + (args: CheckCategoryArgs): void; +} + +export const checkCategory: CheckCategoryFunction = ({ categories, category, at }) => { + if (categories.of(category) !== undefined) return; + throw new ConfigurationError( + `${at} is marked personal in the category "${category}", which app/personal-data.ts does not declare`, + ); +}; + +export interface FormatRetentionFunction { + (retention: Retention): string; +} + +const UNITS: readonly (readonly [string, number])[] = [ + ["d", 86_400_000], + ["h", 3_600_000], + ["m", 60_000], + ["s", 1_000], +]; + +/** + * A retention as configuration writes it, in the largest unit that divides it. + */ +export const formatRetention: FormatRetentionFunction = ({ months, ms }) => { + if (months > 0) return months % 12 === 0 ? `${months / 12}y` : `${months}mo`; + for (const [unit, size] of UNITS) { + if (ms % size === 0) return `${ms / size}${unit}`; + } + return `${ms}ms`; +}; + +interface Marked { + readonly personal: { readonly categories: Readonly> } | null; +} + +export interface CheckMarkedCategoriesArgs { + readonly categories: PersonalCategories; + readonly aggregates: { + readonly byName: Readonly< + Record< + string, + { + readonly eventsByType: Readonly>; + readonly commands: Readonly>; + } + > + >; + }; + readonly processes: { readonly all: readonly ({ readonly name: string } & Marked)[] }; + readonly readModels: Readonly< + Record< + string, + { readonly fields: Readonly> } + > + >; +} + +export interface CheckMarkedCategoriesFunction { + (args: CheckMarkedCategoriesArgs): void; +} + +/** + * Throws `ConfigurationError` for a field marked personal in a category the app does not declare, + * which the types refuse, for a project that skipped the type check. + */ +export const checkMarkedCategories: CheckMarkedCategoriesFunction = ({ + categories, + aggregates, + processes, + readModels, +}) => { + const check = (marked: Marked, at: string): void => { + for (const [field, category] of Object.entries(marked.personal?.categories ?? {})) { + checkCategory({ categories, category, at: `${at}, field "${field}"` }); + } + }; + for (const [name, aggregate] of Object.entries(aggregates.byName)) { + for (const [type, event] of Object.entries(aggregate.eventsByType)) { + check(event, `Event ${name}.${type}`); + } + for (const [type, command] of Object.entries(aggregate.commands)) { + check(command, `Command ${name}.${type}`); + } + } + for (const process of processes.all) check(process, `State of process ${process.name}`); + for (const [name, readModel] of Object.entries(readModels)) { + for (const [field, { category }] of Object.entries(readModel.fields)) { + if (category === null) continue; + checkCategory({ categories, category, at: `Read model ${name}, field "${field}"` }); + } + } +}; diff --git a/packages/core/src/kernel/personal-data/commands.test.ts b/packages/core/src/kernel/personal-data/commands.test.ts index e61d86af..2a1215b2 100644 --- a/packages/core/src/kernel/personal-data/commands.test.ts +++ b/packages/core/src/kernel/personal-data/commands.test.ts @@ -1,5 +1,9 @@ import { describe, expect, it } from "vitest"; import { z } from "zod"; +import { + DEFAULT_PERSONAL_CATEGORY, + type SealedValue, +} from "../../adapter/storage/personal-key-store.ts"; import { DeadLetterNotRetriableError, ValidationError } from "../../contracts/errors.ts"; import { memory } from "../../memory/index.ts"; import type { PersonalDataForgottenEvent } from "../../modules/bounda-events.ts"; @@ -7,7 +11,7 @@ import type { CommandPayloadArgs, EventPayloadArgs } from "../../modules/payload import { person, personal } from "../../modules/personal.ts"; import type { Registry } from "../../modules/registry.ts"; import { createTestApp, expectForgotten } from "../../testing/index.ts"; -import { createRecordingLogger, keysOver, newStoredKey } from "../test-support.ts"; +import { createRecordingLogger, keysOver, newStoredKey, removeKeys } from "../test-support.ts"; import { openCommandPayload, sealCommandPayload } from "./commands.ts"; import { isSealed } from "./crypto.ts"; import { personalMarksOf } from "./marks.ts"; @@ -143,7 +147,7 @@ describe("sealCommandPayload and openCommandPayload", () => { marks: marks({ customerId: z.string(), email: personal(z.email()) }), keys: keysOver(storage.personalKeys), }); - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); expect(sealed.persons).toEqual([{ person: "c-1", fingerprint: key?.fingerprint }]); const payload = sealed.payload as { customerId: unknown; email: unknown }; expect(payload.customerId).toBe("c-1"); @@ -154,9 +158,50 @@ describe("sealCommandPayload and openCommandPayload", () => { }); }); + it("seals the fields of each category under the person's key of that category", async () => { + const storage = await memory().createStorage(); + const sealed = await sealCommandPayload({ + type: "RemindCustomer", + aggregateId: "c-1", + payload: { email: "ana@example.com", iban: "ES00", name: "Ana" }, + marks: marks({ + email: personal(z.email(), { category: "profile" }), + iban: personal(z.string(), { category: "invoices" }), + name: personal(z.string(), { category: "profile" }), + }), + keys: keysOver(storage.personalKeys), + }); + const profile = await storage.personalKeys.get("c-1", "profile"); + const invoices = await storage.personalKeys.get("c-1", "invoices"); + expect(profile?.fingerprint).not.toBe(invoices?.fingerprint); + expect(await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); + expect(sealed.persons).toEqual([ + { person: "c-1", fingerprint: profile?.fingerprint }, + { person: "c-1", fingerprint: invoices?.fingerprint }, + ]); + const payload = sealed.payload as Record; + expect(payload.email?.$personal).toMatchObject({ + category: "profile", + key: profile?.fingerprint, + }); + expect(payload.iban?.$personal).toMatchObject({ + category: "invoices", + key: invoices?.fingerprint, + }); + await storage.personalKeys.remove("c-1", invoices?.fingerprint ?? ""); + expect(await openCommandPayload({ payload, keys: keysOver(storage.personalKeys) })).toEqual({ + payload: { email: "ana@example.com", iban: null, name: "Ana" }, + forgotten: true, + }); + }); + it("seals under the person a field names, and keeps the key in force", async () => { const storage = await memory().createStorage(); - const kept = await storage.personalKeys.add("f-1", await newStoredKey()); + const kept = await storage.personalKeys.add( + "f-1", + DEFAULT_PERSONAL_CATEGORY, + await newStoredKey(), + ); const sealed = await sealCommandPayload({ type: "IntroduceFriend", aggregateId: "c-1", @@ -165,7 +210,7 @@ describe("sealCommandPayload and openCommandPayload", () => { keys: keysOver(storage.personalKeys), }); expect(sealed.persons).toEqual([{ person: "f-1", fingerprint: kept.fingerprint }]); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); }); it("leaves a payload with nothing to seal as it is, naming no one and keying no one", async () => { @@ -186,7 +231,7 @@ describe("sealCommandPayload and openCommandPayload", () => { }), ).toEqual({ payload, persons: [] }); } - expect(await storage.personalKeys.get("f-1")).toBeNull(); + expect(await storage.personalKeys.get("f-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); }); it.each([undefined, "", 42n])( @@ -234,7 +279,7 @@ describe("sealCommandPayload and openCommandPayload", () => { ), ); } - expect(await storage.personalKeys.get("f-1")).toBeNull(); + expect(await storage.personalKeys.get("f-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); }); it("opens to null, telling it forgotten, a field whose key is gone or was replaced", async () => { @@ -246,12 +291,12 @@ describe("sealCommandPayload and openCommandPayload", () => { marks: marks({ email: personal(z.email()), plan: z.string() }), keys: keysOver(storage.personalKeys), }); - await storage.personalKeys.remove("c-1"); + await removeKeys(storage.personalKeys, "c-1"); const forgotten = { payload: { email: null, plan: "pro" }, forgotten: true }; expect(await openCommandPayload({ payload, keys: keysOver(storage.personalKeys) })).toEqual( forgotten, ); - await storage.personalKeys.add("c-1", await newStoredKey()); + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, await newStoredKey()); expect(await openCommandPayload({ payload, keys: keysOver(storage.personalKeys) })).toEqual( forgotten, ); @@ -291,9 +336,9 @@ describe("sealCommandPayload and openCommandPayload", () => { const reads: string[] = []; const keys = keysOver({ ...storage.personalKeys, - get: (person: string) => { + find: (person: string, fingerprint: string) => { reads.push(person); - return storage.personalKeys.get(person); + return storage.personalKeys.find(person, fingerprint); }, }); expect(await openCommandPayload({ payload: ana, keys })).toEqual({ @@ -323,7 +368,7 @@ describe("personal data in delayed commands", () => { const { app, storage, runDue } = await setup(); await app.commands.remindCustomer(ana, { delay: "1m" }); const [entry] = await storage.scheduler.list(); - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); expect(entry?.persons).toEqual([{ person: "c-1", fingerprint: key?.fingerprint }]); expect(isSealed((entry?.command.payload as { email?: unknown } | undefined)?.email)).toBe(true); expect(JSON.stringify(entry)).not.toContain("ana@example.com"); @@ -373,6 +418,10 @@ describe("personal data in delayed commands", () => { expect(await forgottenPayload()).toEqual({ personId: "c-1", reason: "requested", + whole: true, + categories: [DEFAULT_PERSONAL_CATEGORY], + retained: [], + expired: [], fingerprints: [expect.any(String)], cancelledCommands: named.map((entry) => ({ commandId: entry.dedupeKey.replace(/^command:/, ""), @@ -396,7 +445,7 @@ describe("personal data in delayed commands", () => { await app.commands.remindCustomer(ana, { delay: "1m" }); await runDue(); const [stored] = await storage.deadLetterStore.list(); - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); expect(stored?.persons).toEqual([{ person: "c-1", fingerprint: key?.fingerprint }]); expect(JSON.stringify(stored)).not.toContain("ana@example.com"); const opened = { @@ -418,6 +467,10 @@ describe("personal data in delayed commands", () => { expect(await forgottenPayload()).toEqual({ personId: "c-1", reason: "requested", + whole: true, + categories: [DEFAULT_PERSONAL_CATEGORY], + retained: [], + expired: [], fingerprints: [key?.fingerprint], cancelledCommands: [], removedDeadLetters: [{ id: stored?.id, commandType: "RemindCustomer" }], @@ -467,8 +520,8 @@ describe("personal data in delayed commands", () => { it("drops, never running it, a delayed command sealed under a key its person no longer holds", async () => { const { app, storage, logs, runDue } = await setup(); await app.commands.remindCustomer(ana, { delay: "1m" }); - await storage.personalKeys.remove("c-1"); - await storage.personalKeys.add("c-1", await newStoredKey()); + await removeKeys(storage.personalKeys, "c-1"); + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, await newStoredKey()); await runDue(); expect(received).toEqual([]); expect(await storage.scheduler.list()).toEqual([]); @@ -487,7 +540,7 @@ describe("personal data in delayed commands", () => { await app.commands.remindCustomer(ana, { delay: "1m" }); await runDue(); failing = false; - await storage.personalKeys.remove("c-1"); + await removeKeys(storage.personalKeys, "c-1"); const [letter] = await app.deadLetters.list(); expect(letter?.payload).toEqual({ customerId: "c-1", email: null }); await expect(app.deadLetters.retry(letter?.id ?? "")).rejects.toThrow( diff --git a/packages/core/src/kernel/personal-data/commands.ts b/packages/core/src/kernel/personal-data/commands.ts index 1d7b0114..4a94f91e 100644 --- a/packages/core/src/kernel/personal-data/commands.ts +++ b/packages/core/src/kernel/personal-data/commands.ts @@ -1,8 +1,8 @@ -import type { KeyedPerson } from "../../adapter/storage/personal-key-store.ts"; +import { type KeyedPerson, keyedPersonId } from "../../adapter/storage/personal-key-store.ts"; import { ValidationError } from "../../contracts/errors.ts"; import { isRecord } from "../shared/merge-fields.ts"; import { isSealed, type PersonalKey, refuseSealed, seal, unseal } from "./crypto.ts"; -import type { PersonalMarks } from "./marks.ts"; +import { fieldCategory, type PersonalMarks } from "./marks.ts"; import type { PersonalKeys } from "./wrapped-keys.ts"; export interface SealCommandPayloadArgs { @@ -40,8 +40,12 @@ const personOf = ( return String(value); }; -const keyInForce = async (keys: PersonalKeys, person: string): Promise => - (await keys.held(person)) ?? (await keys.create(person)); +const keyInForce = async ( + keys: PersonalKeys, + person: string, + category: string, +): Promise => + (await keys.held(person, category)) ?? (await keys.create(person, category)); export interface KeyOfFunction { (person: string, fingerprint: string): Promise; @@ -52,29 +56,26 @@ export interface SealedKeysFunction { } /** - * Reads the key a value was sealed under, `null` when the person holds none or another one: - * each person's key read and imported once, whatever number of values it opens. + * Reads the key a value was sealed under, `null` once the person no longer holds it: each key + * read and imported once, whatever number of values it opens. */ export const sealedKeys: SealedKeysFunction = (keys) => { const imported = new Map>(); return (person, fingerprint) => { - const id = `${person}\u0000${fingerprint}`; + const id = keyedPersonId({ person, fingerprint }); const known = imported.get(id); if (known !== undefined) return known; - const key = keys - .get(person) - .then((stored) => - stored === null || stored.fingerprint !== fingerprint ? null : keys.open(person, stored), - ); + const key = keys.opened(person, fingerprint); imported.set(id, key); return key; }; }; /** - * Seals the payload's personal fields with the key of the person the command names, created on - * their first sealed field. A key removed before the payload is stored leaves it sealed under a key - * no one holds: the command then never runs, as `openCommandPayload` tells. + * Seals the payload's personal fields with the person's key of each field's category, the person + * the command names, created on their first sealed field. A key removed before the payload is + * stored leaves it sealed under a key no one holds: the command then never runs, as + * `openCommandPayload` tells. */ export const sealCommandPayload: SealCommandPayloadFunction = async (args) => { const { payload, marks, keys } = args; @@ -84,12 +85,15 @@ export const sealCommandPayload: SealCommandPayloadFunction = async (args) => { const present = marks.fields.filter((field) => payload[field] !== undefined); if (present.length === 0) return { payload, persons: [] }; const person = personOf(args, payload, marks.person); - const key = await keyInForce(keys, person); const sealed: Record = { ...payload }; + const persons = new Map(); for (const field of present) { - sealed[field] = await seal({ key, person, value: payload[field] }); + const category = fieldCategory(marks, field); + const key = await keyInForce(keys, person, category); + persons.set(key.fingerprint, { person, fingerprint: key.fingerprint }); + sealed[field] = await seal({ key, person, category, value: payload[field] }); } - return { payload: sealed, persons: [{ person, fingerprint: key.fingerprint }] }; + return { payload: sealed, persons: [...persons.values()] }; }; export interface OpenCommandPayloadArgs { diff --git a/packages/core/src/kernel/personal-data/crypto.test.ts b/packages/core/src/kernel/personal-data/crypto.test.ts index 285f8907..7a0cdae3 100644 --- a/packages/core/src/kernel/personal-data/crypto.test.ts +++ b/packages/core/src/kernel/personal-data/crypto.test.ts @@ -1,5 +1,7 @@ import { afterEach, describe, expect, it, vi } from "vitest"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { + categoryOf, createDisposableKey, fingerprintOf, generateWrappableKey, @@ -18,7 +20,7 @@ describe("personal data crypto", () => { it("seals a value under a person and opens it with the same key, whatever its type", async () => { const key = await createDisposableKey(); for (const value of ["ana@example.com", 42, true, null, { street: "Main 1" }, ["a", "b"]]) { - const sealed = await seal({ key, person: "c-1", value }); + const sealed = await seal({ key, person: "c-1", category: DEFAULT_PERSONAL_CATEGORY, value }); expect(isSealed(sealed)).toBe(true); expect(sealed.$personal.person).toBe("c-1"); expect(sealed.$personal.key).toBe(key.fingerprint); @@ -26,11 +28,22 @@ describe("personal data crypto", () => { } }); + it("names the category in the envelope only when it is not the default one", async () => { + const key = await createDisposableKey(); + const plain = await seal({ key, person: "c-1", category: DEFAULT_PERSONAL_CATEGORY, value: 1 }); + expect(plain.$personal).not.toHaveProperty("category"); + expect(categoryOf(plain)).toBe(DEFAULT_PERSONAL_CATEGORY); + const invoices = await seal({ key, person: "c-1", category: "invoices", value: 1 }); + expect(invoices.$personal.category).toBe("invoices"); + expect(categoryOf(invoices)).toBe("invoices"); + expect(await unseal({ key, sealed: invoices })).toBe(1); + }); + it("leaves no trace of the value in the sealed text", async () => { const key = await createDisposableKey(); // Long enough that random base64 cannot spell them by chance, as it can `42` or `null`. for (const value of ["ana@example.com", { street: "Main Street 1" }]) { - const sealed = await seal({ key, person: "c-1", value }); + const sealed = await seal({ key, person: "c-1", category: DEFAULT_PERSONAL_CATEGORY, value }); expect(sealed.$personal.data).not.toContain(JSON.stringify(value)); } }); @@ -40,8 +53,8 @@ describe("personal data crypto", () => { expect(one.fingerprint).not.toBe(other.fingerprint); expect(one.fingerprint).toHaveLength(24); const [first, second] = await Promise.all([ - seal({ key: one, person: "c-1", value: "same" }), - seal({ key: one, person: "c-1", value: "same" }), + seal({ key: one, person: "c-1", category: DEFAULT_PERSONAL_CATEGORY, value: "same" }), + seal({ key: one, person: "c-1", category: DEFAULT_PERSONAL_CATEGORY, value: "same" }), ]); expect(first.$personal.iv).not.toBe(second.$personal.iv); expect(first.$personal.data).not.toBe(second.$personal.data); @@ -68,7 +81,12 @@ describe("personal data crypto", () => { const key = await createDisposableKey(); const firsts = new Set(); for (let sealing = 0; sealing < 64; sealing += 1) { - const { $personal } = await seal({ key, person: "c-1", value: sealing }); + const { $personal } = await seal({ + key, + person: "c-1", + category: DEFAULT_PERSONAL_CATEGORY, + value: sealing, + }); const first = atob($personal.iv).charCodeAt(0); expect(first).toBeGreaterThanOrEqual(0x80); firsts.add(first); @@ -135,7 +153,12 @@ describe("personal data crypto", () => { it("opens nothing without a key or with another one, and refuses a value its key should open", async () => { const [key, other] = await Promise.all([createDisposableKey(), createDisposableKey()]); - const sealed = await seal({ key, person: "c-1", value: "ana@example.com" }); + const sealed = await seal({ + key, + person: "c-1", + category: DEFAULT_PERSONAL_CATEGORY, + value: "ana@example.com", + }); expect(await unseal({ key: null, sealed })).toBeNull(); expect(await unseal({ key: other, sealed })).toBeNull(); const forged = { $personal: { ...sealed.$personal, data: sealed.$personal.iv } }; @@ -145,6 +168,9 @@ describe("personal data crypto", () => { it("takes for sealed only an object holding nothing but a complete envelope", () => { const envelope = { person: "c-1", key: "k", iv: "aQ==", data: "ZA==" }; expect(isSealed({ $personal: envelope })).toBe(true); + expect(isSealed({ $personal: { ...envelope, category: "invoices" } })).toBe(true); + expect(isSealed({ $personal: { ...envelope, category: 1 } })).toBe(false); + expect(isSealed({ $personal: { ...envelope, category: null } })).toBe(false); expect(isSealed({ $personal: envelope, more: 1 })).toBe(false); expect(isSealed({ $personal: { ...envelope, data: 1 } })).toBe(false); expect(isSealed({ $personal: { person: "c-1", iv: "aQ==" } })).toBe(false); diff --git a/packages/core/src/kernel/personal-data/crypto.ts b/packages/core/src/kernel/personal-data/crypto.ts index 1b7072fa..176a95ad 100644 --- a/packages/core/src/kernel/personal-data/crypto.ts +++ b/packages/core/src/kernel/personal-data/crypto.ts @@ -1,4 +1,8 @@ -import type { PersonalEnvelope, SealedValue } from "../../adapter/storage/personal-key-store.ts"; +import { + DEFAULT_PERSONAL_CATEGORY, + type PersonalEnvelope, + type SealedValue, +} from "../../adapter/storage/personal-key-store.ts"; import { ValidationError } from "../../contracts/errors.ts"; export type { PersonalEnvelope, SealedValue }; @@ -8,12 +12,14 @@ const KEY_BITS = 256; const IV_BYTES = 12; /** - * A person's key as the kernel seals and opens with it, with the fingerprint envelopes name it by. - * The key can never be exported: its raw bytes are not reachable from JavaScript. + * A person's key as the kernel seals and opens with it, with the fingerprint envelopes name it by, + * and whether a forgetting kept it past it, as the store said when it was read. The key can never + * be exported: its raw bytes are not reachable from JavaScript. */ export interface PersonalKey { readonly fingerprint: string; readonly cryptoKey: CryptoKey; + readonly retained: boolean; } export interface IsSealedFunction { @@ -30,6 +36,7 @@ export const isSealed: IsSealedFunction = (value): value is SealedValue => { typeof envelope === "object" && envelope !== null && typeof Reflect.get(envelope, "person") === "string" && + ["string", "undefined"].includes(typeof Reflect.get(envelope, "category")) && typeof Reflect.get(envelope, "key") === "string" && typeof Reflect.get(envelope, "iv") === "string" && typeof Reflect.get(envelope, "data") === "string" @@ -152,7 +159,7 @@ export const createDisposableKey: CreateDisposableKeyFunction = async () => { false, [...PERSONAL_KEY_USAGES], ); - return { fingerprint: await fingerprintOf(cryptoKey), cryptoKey }; + return { fingerprint: await fingerprintOf(cryptoKey), cryptoKey, retained: false }; }; export interface ImportRawKeyArgs { @@ -198,6 +205,7 @@ export const withRawKey: WithRawKeyFunction = async (key, use) => { export interface SealArgs { readonly key: PersonalKey; readonly person: string; + readonly category: string; readonly value: unknown; } @@ -207,9 +215,10 @@ export interface SealFunction { /** * Encrypts `value` as JSON, under a nonce of its own, so equal values never store alike, and never - * the fingerprint's. + * the fingerprint's. The envelope names the category unless it is the default one, as every + * envelope sealed before categories existed reads. */ -export const seal: SealFunction = async ({ key, person, value }) => { +export const seal: SealFunction = async ({ key, person, category, value }) => { const iv = crypto.getRandomValues(new Uint8Array(IV_BYTES)); iv[0] = (iv[0] ?? 0) | SEAL_IV_MARK; const encrypted = await crypto.subtle.encrypt( @@ -220,6 +229,7 @@ export const seal: SealFunction = async ({ key, person, value }) => { return { $personal: { person, + ...(category === DEFAULT_PERSONAL_CATEGORY ? {} : { category }), key: key.fingerprint, iv: toBase64(iv), data: toBase64(new Uint8Array(encrypted)), @@ -236,6 +246,13 @@ export interface UnsealFunction { (args: UnsealArgs): Promise; } +export interface CategoryOfFunction { + (sealed: SealedValue): string; +} + +export const categoryOf: CategoryOfFunction = ({ $personal }) => + $personal.category ?? DEFAULT_PERSONAL_CATEGORY; + /** * `null` without a key, or with one the value was not sealed under: the person was forgotten, * and may have been given a new key since. A value the key should open and does not throws. diff --git a/packages/core/src/kernel/personal-data/dead-letters.test.ts b/packages/core/src/kernel/personal-data/dead-letters.test.ts index 19d9f830..55c2e161 100644 --- a/packages/core/src/kernel/personal-data/dead-letters.test.ts +++ b/packages/core/src/kernel/personal-data/dead-letters.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from "vitest"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { ValidationError } from "../../contracts/errors.ts"; import { silentLogger } from "../../contracts/logger.ts"; import { memory } from "../../memory/index.ts"; @@ -311,7 +312,7 @@ describe("dead letters of work that names a person", () => { const stored = await storage.deadLetterStore.list(); expect(stored).toHaveLength(2); expect(JSON.stringify(stored)).not.toContain("ana@example.com"); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); // The handler that failed first read the email; the other read the event after the forget. const listed = await app.deadLetters.list(); for (const [index, letter] of stored.entries()) { @@ -334,7 +335,7 @@ describe("dead letters of work that names a person", () => { }; await app.commands.registerCustomer(ana); await app.runUntilIdle(); - expect(await storage.personalKeys.get("c-1")).not.toBeNull(); + expect(await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).not.toBeNull(); const listed = await app.deadLetters.list(); expect(JSON.stringify(listed)).not.toContain("ana@example.com"); // The policy read the first registration under the key the forget removed, and the second @@ -386,7 +387,7 @@ describe("dead letters of work that names a person", () => { expect(delayed).toMatchObject({ person: "c-1" }); expect(isSealed(delayed?.errorMessage)).toBe(true); expect(JSON.stringify(delayed)).not.toContain("ana@example.com"); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); await app.stop(); }); }); diff --git a/packages/core/src/kernel/personal-data/errors.test.ts b/packages/core/src/kernel/personal-data/errors.test.ts index 16bd61d8..0645d678 100644 --- a/packages/core/src/kernel/personal-data/errors.test.ts +++ b/packages/core/src/kernel/personal-data/errors.test.ts @@ -1,14 +1,15 @@ import { describe, expect, it } from "vitest"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { ValidationError } from "../../contracts/errors.ts"; import { memory } from "../../memory/index.ts"; import { keysOver, newStoredKey } from "../test-support.ts"; import { sealedKeys } from "./commands.ts"; -import { isSealed } from "./crypto.ts"; -import { openTextWith, sealError } from "./errors.ts"; +import { isSealed, unseal } from "./crypto.ts"; +import { openTextWith, sealError, sealUnderAll } from "./errors.ts"; const setup = async () => { const keys = keysOver((await memory().createStorage()).personalKeys); - const held = await keys.add("c-1", await newStoredKey()); + const held = await keys.add("c-1", DEFAULT_PERSONAL_CATEGORY, await newStoredKey()); return { keys, held, open: () => openTextWith(sealedKeys(keys)) }; }; @@ -51,20 +52,100 @@ describe("sealError", () => { }); it("reads null once the person is forgotten", async () => { - const { keys, open } = await setup(); + const { keys, held, open } = await setup(); const before = await sealError({ error: new Error("first"), person: "c-1", keys }); - await keys.remove("c-1"); + await keys.remove("c-1", held.fingerprint); expect(await open()(before.errorMessage)).toBeNull(); expect(await open()(before.errorStack ?? "")).toBeNull(); }); it("never gives a forgotten person a key, and seals what their work read unreadable", async () => { - const { keys, open } = await setup(); - await keys.remove("c-1"); + const { keys, held, open } = await setup(); + await keys.remove("c-1", held.fingerprint); const sealed = await sealError({ error: new Error("ana@example.com"), person: "c-1", keys }); - expect(await keys.get("c-1")).toBeNull(); + expect(await keys.list("c-1")).toEqual([]); expect(isSealed(sealed.errorMessage)).toBe(true); expect(JSON.stringify(sealed)).not.toContain("ana@example.com"); expect(await open()(sealed.errorMessage)).toBeNull(); }); + + it("seals the text under every key the person holds, one inside the other by category, opening only while they hold all", async () => { + const { keys, held, open } = await setup(); + const invoices = await keys.add("c-1", "invoices", await newStoredKey()); + const profile = await keys.add("c-1", "profile", await newStoredKey()); + await keys.retain("c-1", invoices.fingerprint, { + since: "2026-01-01T00:00:00.000Z", + lastActiveAt: "2025-12-01T00:00:00.000Z", + }); + const sealed = await sealError({ error: new Error("ana@example.com"), person: "c-1", keys }); + const layers: string[] = []; + let layer: unknown = sealed.errorMessage; + while (isSealed(layer)) { + layers.push(layer.$personal.key); + const key = await keys.opened("c-1", layer.$personal.key); + layer = await unseal({ key, sealed: layer }); + } + expect(layers).toEqual([profile.fingerprint, invoices.fingerprint, held.fingerprint]); + expect(layer).toBe("ana@example.com"); + expect(await open()(sealed.errorMessage)).toBe("ana@example.com"); + expect(await open()(sealed.errorStack ?? "")).toContain("ana@example.com"); + await keys.remove("c-1", invoices.fingerprint); + expect(await open()(sealed.errorMessage)).toBeNull(); + expect(await open()(sealed.errorStack ?? "")).toBeNull(); + }); + + it("seals under a key no store keeps what the work read under a key the person no longer holds", async () => { + const { keys, held, open } = await setup(); + const read = [ + { person: "c-1", fingerprint: "gone" }, + { person: "c-2", fingerprint: "other" }, + ]; + const sealed = await sealError({ + error: new Error("ana@example.com"), + person: "c-1", + keys, + read, + }); + expect(isSealed(sealed.errorMessage)).toBe(true); + expect(sealed.errorMessage).not.toMatchObject({ $personal: { key: held.fingerprint } }); + expect(await open()(sealed.errorMessage)).toBeNull(); + const readHeld = [{ person: "c-1", fingerprint: held.fingerprint }]; + const kept = await sealError({ error: new Error("bo"), person: "c-1", keys, read: readHeld }); + expect(await open()(kept.errorMessage)).toBe("bo"); + }); + + it("seals under a key no store keeps what the work read under a key a forgetting kept since", async () => { + const { keys, held, open } = await setup(); + await keys.retain("c-1", held.fingerprint, { + since: "2026-02-01T00:00:00.000Z", + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + const readInForce = [{ person: "c-1", fingerprint: held.fingerprint }]; + const dropped = await sealError({ + error: new Error("ana@example.com"), + person: "c-1", + keys, + read: readInForce, + }); + expect(await open()(dropped.errorMessage)).toBeNull(); + const readKept = [{ person: "c-1", fingerprint: held.fingerprint, retained: true as const }]; + const kept = await sealError({ error: new Error("bo"), person: "c-1", keys, read: readKept }); + expect(await open()(kept.errorMessage)).toBe("bo"); + }); +}); + +describe("sealUnderAll", () => { + it("seals under each key in turn, the last one outermost", async () => { + const { keys, held } = await setup(); + const other = await keys.add("c-1", "invoices", await newStoredKey()); + const under = await keys.heldAll("c-1"); + const sealed = await sealUnderAll(under, "c-1", "text"); + const last = under.at(-1); + expect(sealed.$personal).toMatchObject({ person: "c-1", key: last?.key.fingerprint }); + expect(under.map(({ key }) => key.fingerprint).toSorted()).toEqual( + [held.fingerprint, other.fingerprint].toSorted(), + ); + const opened = await openTextWith(sealedKeys(keys))(sealed); + expect(opened).toBe("text"); + }); }); diff --git a/packages/core/src/kernel/personal-data/errors.ts b/packages/core/src/kernel/personal-data/errors.ts index d61e2ae0..e835e70a 100644 --- a/packages/core/src/kernel/personal-data/errors.ts +++ b/packages/core/src/kernel/personal-data/errors.ts @@ -1,9 +1,10 @@ import type { KeyedPerson, SealedValue } from "../../adapter/storage/personal-key-store.ts"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { type ErrorIdentity, errorDetails, errorIdentity } from "../shared/retry.ts"; import type { KeyOfFunction } from "./commands.ts"; -import { createDisposableKey, type PersonalKey, seal, unseal } from "./crypto.ts"; +import { createDisposableKey, isSealed, seal, unseal } from "./crypto.ts"; import { readUnder } from "./reads.ts"; -import type { PersonalKeys } from "./wrapped-keys.ts"; +import { type CategoryKey, inSealingOrder, type PersonalKeys } from "./wrapped-keys.ts"; export interface SealErrorArgs { readonly error: unknown; @@ -20,8 +21,8 @@ export interface SealErrorArgs { } /** - * An error as a dead letter records it: its identity in the clear, its text sealed under the - * person the failed work names. + * An error as a dead letter records it: its identity in the clear, its text sealed under the keys + * of the person the failed work names. */ export interface SealedError extends ErrorIdentity { readonly errorMessage: string | SealedValue; @@ -32,25 +33,44 @@ export interface SealErrorFunction { (args: SealErrorArgs): Promise; } -// The person holds no key, or not every key the work read their data under: they were forgotten -// since, and maybe keyed anew. A key no store keeps seals the text unreadable, as the forget would -// have, where the new key would open what the work read before it. -const keyToSealWith = async ( +// The person holds no key, or not every key the work read their data under as it read it: they +// were forgotten since, and maybe keyed anew, or a forgetting kept a key read in force. A key no +// store keeps seals the text unreadable, as the forget would have, where a new or kept key would +// open what the work read before it. +const keysToSealWith = async ( keys: PersonalKeys, person: string, - readUnder: readonly string[], -): Promise => { - const held = await keys.get(person); - return held !== null && readUnder.every((fingerprint) => fingerprint === held.fingerprint) - ? keys.open(person, held) - : createDisposableKey(); + readUnder: readonly KeyedPerson[], +): Promise => { + const held = await keys.heldAll(person); + const holds = ({ fingerprint, retained }: KeyedPerson): boolean => + held.some(({ key }) => key.fingerprint === fingerprint && (!key.retained || retained === true)); + return held.length > 0 && readUnder.every(holds) + ? inSealingOrder(held) + : [{ category: DEFAULT_PERSONAL_CATEGORY, key: await createDisposableKey() }]; +}; + +export interface SealUnderAllFunction { + (under: readonly CategoryKey[], person: string, value: string): Promise; +} + +/** + * Text sealed under each key in turn, each sealing what the one before sealed, so it opens only + * while the person holds every one of them. + */ +export const sealUnderAll: SealUnderAllFunction = async (under, person, value) => { + let sealed: unknown = value; + for (const { category, key } of under) + sealed = await seal({ key, person, category, value: sealed }); + return sealed as SealedValue; }; /** - * Seals the message and the stack with the key the person holds, never creating one, and only - * when the work read nothing of theirs under another: a handler's error may quote what it read of - * the person, so no store keeps its text in the clear once the work names someone, and a forget - * that came meanwhile still reaches it. + * Seals the message and the stack under every key the person holds, never creating one, and only + * when the work read nothing of theirs under a key they no longer hold: a handler's error may quote + * what it read of the person, in any category, so no store keeps its text in the clear once the + * work names someone, and forgetting any category of theirs, as a forget that came meanwhile, + * still reaches it. */ export const sealError: SealErrorFunction = async ({ error, person, keys, read = [] }) => { const details = errorDetails(error); @@ -62,13 +82,13 @@ export const sealError: SealErrorFunction = async ({ error, person, keys, read = ...(details.stack === undefined ? {} : { errorStack: details.stack }), }; } - const key = await keyToSealWith(keys, person, readUnder({ person, error, also: read })); + const under = await keysToSealWith(keys, person, readUnder({ person, error, also: read })); return { ...identity, - errorMessage: await seal({ key, person, value: details.message }), + errorMessage: await sealUnderAll(under, person, details.message), ...(details.stack === undefined ? {} - : { errorStack: await seal({ key, person, value: details.stack }) }), + : { errorStack: await sealUnderAll(under, person, details.stack) }), }; }; @@ -84,10 +104,18 @@ export interface OpenTextWithFunction { (keyOf: KeyOfFunction): OpenTextFunction; } -export const openTextWith: OpenTextWithFunction = (keyOf) => async (value) => - typeof value === "string" - ? value - : ((await unseal({ - key: await keyOf(value.$personal.person, value.$personal.key), - sealed: value, - })) as string | null); +/** + * Opens text sealed under one key or under several, one inside the other, `null` as soon as one + * of them is gone. + */ +export const openTextWith: OpenTextWithFunction = (keyOf) => { + const open = async (value: string | SealedValue): Promise => { + if (typeof value === "string") return value; + const opened = await unseal({ + key: await keyOf(value.$personal.person, value.$personal.key), + sealed: value, + }); + return isSealed(opened) ? open(opened) : (opened as string | null); + }; + return open; +}; diff --git a/packages/core/src/kernel/personal-data/event-store.test.ts b/packages/core/src/kernel/personal-data/event-store.test.ts index 13f1b404..4f5c8971 100644 --- a/packages/core/src/kernel/personal-data/event-store.test.ts +++ b/packages/core/src/kernel/personal-data/event-store.test.ts @@ -1,7 +1,12 @@ import { describe, expect, it } from "vitest"; import { ForgottenSinceLoadError } from "../../adapter/storage/append-checks.ts"; import type { AppendArgs } from "../../adapter/storage/event-store.ts"; -import type { KeyedPerson, PersonalKeyStore } from "../../adapter/storage/personal-key-store.ts"; +import { + DEFAULT_PERSONAL_CATEGORY, + type KeyedPerson, + type PersonalKeyStore, + type SealedValue, +} from "../../adapter/storage/personal-key-store.ts"; import { pendingEvent } from "../../adapter/testing/fixtures.ts"; import { ConcurrencyError, @@ -15,7 +20,7 @@ import type { Registry } from "../../modules/registry.ts"; import { buildAggregates } from "../aggregate/build-aggregates.ts"; import { withUpcasting } from "../aggregate/upcasting.ts"; import type { ProcessesRuntime } from "../process/build-processes.ts"; -import { keysOver, newStoredKey } from "../test-support.ts"; +import { keysOver, newStoredKey, removeKeys } from "../test-support.ts"; import { isSealed } from "./crypto.ts"; import { openedUnder, withPersonalData } from "./event-store.ts"; @@ -29,6 +34,15 @@ const registry: Registry = { evolve: () => ({}), }, customerRenamed: { evolve: () => ({}) }, + customerBilled: { + payload: ({ z, personal }: EventPayloadArgs) => + z.object({ + name: personal(z.string(), { category: "profile" }), + iban: personal(z.string(), { category: "invoices" }), + email: personal(z.email()).optional(), + }), + evolve: () => ({}), + }, }, commands: {}, policies: {}, @@ -59,21 +73,19 @@ const setup = async () => { const storage = await memory().createStorage(); const calls: string[] = []; const keys: PersonalKeyStore = { - get: async (person) => { + ...storage.personalKeys, + get: async (person, category) => { calls.push(`get:${person}`); - return storage.personalKeys.get(person); + return storage.personalKeys.get(person, category); }, - add: async (person, key) => { + find: async (person, fingerprint) => { + calls.push(`find:${person}`); + return storage.personalKeys.find(person, fingerprint); + }, + add: async (person, category, key) => { calls.push(`add:${person}`); - return storage.personalKeys.add(person, key); + return storage.personalKeys.add(person, category, key); }, - remove: (person) => storage.personalKeys.remove(person), - listWrappedUnder: (args) => storage.personalKeys.listWrappedUnder(args), - rewrap: (person, from, to) => storage.personalKeys.rewrap(person, from, to), - hold: (person, from, hold) => storage.personalKeys.hold(person, from, hold), - nextHoldEnd: (args) => storage.personalKeys.nextHoldEnd(args), - countWrappedUnder: (args) => storage.personalKeys.countWrappedUnder(args), - releaseHolds: () => storage.personalKeys.releaseHolds(), }; const store = withPersonalData({ eventStore: storage.eventStore, @@ -110,7 +122,9 @@ describe("withPersonalData", () => { expect(isSealed(payload.email)).toBe(true); expect(JSON.stringify(stored)).not.toContain("ana@example.com"); expect((payload.email as { $personal: { person: string } }).$personal.person).toBe("c-1"); - expect((await storage.personalKeys.get("c-1"))?.fingerprint).toHaveLength(24); + expect( + (await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY))?.fingerprint, + ).toHaveLength(24); const loaded = await store.load({ aggregateType: "customer", aggregateId: "c-1" }); expect(loaded.events[0]?.payload).toEqual({ email: "ana@example.com", plan: "pro" }); expect((await store.readAll({ afterPosition: 0, limit: 10 }))[0]?.payload).toEqual({ @@ -139,8 +153,10 @@ describe("withPersonalData", () => { expect(payload.customerId).toBe("c-7"); expect(isSealed(payload.address)).toBe(true); expect((payload.address as { $personal: { person: string } }).$personal.person).toBe("c-7"); - expect((await storage.personalKeys.get("c-7"))?.fingerprint).toHaveLength(24); - expect(await storage.personalKeys.get("o-1")).toBeNull(); + expect( + (await storage.personalKeys.get("c-7", DEFAULT_PERSONAL_CATEGORY))?.fingerprint, + ).toHaveLength(24); + expect(await storage.personalKeys.get("o-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); const loaded = await store.load({ aggregateType: "order", aggregateId: "o-1" }); expect(loaded.events[0]?.payload).toEqual({ customerId: "c-7", @@ -165,14 +181,14 @@ describe("withPersonalData", () => { placed({ customerId: "c-1", note: "ring twice", address: { street: "x" }, total: 1 }), ], }); - await storage.personalKeys.remove("c-1"); + await removeKeys(storage.personalKeys, "c-1"); const all = await store.readAll({ afterPosition: 0, limit: 10 }); expect(all.map((event) => event.payload)).toEqual([ { email: null, plan: "pro" }, { customerId: "c-1", note: null, address: null, total: 1 }, ]); const loaded = await store.load({ aggregateType: "customer", aggregateId: "c-1" }); - expect(loaded.fingerprints).toEqual(new Map()); + expect(loaded.opened).toEqual([]); }); it("reads what was sealed under a key the person no longer has as null, next to what their new key opens", async () => { @@ -183,7 +199,7 @@ describe("withPersonalData", () => { expectedVersion: 0, events: [registered(1, "ana@example.com")], }); - await storage.personalKeys.remove("c-1"); + await removeKeys(storage.personalKeys, "c-1"); await store.append({ aggregateType: "customer", aggregateId: "c-1", @@ -195,8 +211,8 @@ describe("withPersonalData", () => { null, "ana@again.example", ]); - const again = await storage.personalKeys.get("c-1"); - expect(loaded.fingerprints).toEqual(new Map([["c-1", again?.fingerprint]])); + const again = await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); + expect(loaded.opened).toEqual([{ person: "c-1", fingerprint: again?.fingerprint }]); }); it("seals a person the events were decided on only under the key they were read under", async () => { @@ -207,13 +223,10 @@ describe("withPersonalData", () => { expectedVersion: 0, events: [registered(1, "ana@example.com")], }); - const { fingerprints } = await store.load({ aggregateType: "customer", aggregateId: "c-1" }); + const { opened = [] } = await store.load({ aggregateType: "customer", aggregateId: "c-1" }); const other = await newStoredKey(); - await storage.personalKeys.add("c-2", other); - const read = [ - { person: "c-2", fingerprint: other.fingerprint }, - { person: "c-1", fingerprint: fingerprints?.get("c-1") ?? "" }, - ]; + await storage.personalKeys.add("c-2", DEFAULT_PERSONAL_CATEGORY, other); + const read = [{ person: "c-2", fingerprint: other.fingerprint }, ...opened]; const append = (version: number, persons: readonly KeyedPerson[]) => store.append({ aggregateType: "customer", @@ -223,10 +236,10 @@ describe("withPersonalData", () => { persons, }); await append(2, read); - await storage.personalKeys.remove("c-1"); + await removeKeys(storage.personalKeys, "c-1"); calls.length = 0; const refused = await append(3, read).catch((error: unknown) => error); - expect(refused).toBeInstanceOf(ConcurrencyError); + expect(refused).toBeInstanceOf(ForgottenSinceLoadError); expect(refused).toMatchObject({ code: "CONCURRENCY_CONFLICT", streamId: "customer:c-1", @@ -234,26 +247,21 @@ describe("withPersonalData", () => { actualVersion: 2, message: "Person c-1 was forgotten since customer:c-1 was loaded at version 2", }); - expect(calls).toEqual(["get:c-1"]); - expect(await storage.personalKeys.get("c-1")).toBeNull(); - - const fresh = await newStoredKey(); - await storage.personalKeys.add("c-1", fresh); - await expect(append(3, read)).rejects.toBeInstanceOf(ConcurrencyError); - await expect( - append(3, [...read, { person: "c-1", fingerprint: fresh.fingerprint }]), - ).rejects.toBeInstanceOf(ConcurrencyError); - expect(await storage.personalKeys.get("c-1")).toEqual(fresh); - expect( - (await storage.eventStore.load({ aggregateType: "customer", aggregateId: "c-1" })).version, - ).toBe(2); - - await append(3, [{ person: "c-1", fingerprint: fresh.fingerprint }, ...read.slice(0, 1)]); - await append(4, [{ person: "c-1", fingerprint: null }]); - await storage.personalKeys.remove("c-1"); + expect(calls).toEqual(["find:c-1"]); + expect(await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); + await append(3, [{ person: "c-1", fingerprint: null }]); + const fresh = await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); + expect(fresh).not.toBeNull(); + expect(read.map(({ fingerprint }) => fingerprint)).not.toContain(fresh?.fingerprint); + await expect(append(4, read)).rejects.toBeInstanceOf(ConcurrencyError); + await append(4, [ + { person: "c-1", fingerprint: fresh?.fingerprint ?? null }, + ...read.slice(0, 1), + ]); + await removeKeys(storage.personalKeys, "c-1"); await append(5, [{ person: "c-1", fingerprint: null }]); - const created = await storage.personalKeys.get("c-1"); - expect(created?.fingerprint).not.toBe(fresh.fingerprint); + const renewed = await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); + expect(renewed?.fingerprint).not.toBe(fresh?.fingerprint); const loaded = await store.load({ aggregateType: "customer", aggregateId: "c-1" }); expect(loaded.events.map((event) => (event.payload as { email: unknown }).email)).toEqual([ null, @@ -287,7 +295,7 @@ describe("withPersonalData", () => { "ana@other.example", "ana@third.example", ]); - expect(calls).toEqual(["get:c-1"]); + expect(calls).toEqual(["find:c-1"]); const persons = new Set( (await storage.eventStore.readAll({ afterPosition: 0, limit: 10 })).map( (event) => @@ -299,12 +307,12 @@ describe("withPersonalData", () => { it("fails as a conflict when the key it sealed under is removed before the store writes", async () => { const { storage, registered } = await setup(); - await storage.personalKeys.add("c-1", await newStoredKey()); + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, await newStoredKey()); const forgetting: PersonalKeyStore = { ...storage.personalKeys, - get: async (person) => { - const read = await storage.personalKeys.get(person); - await storage.personalKeys.remove(person); + get: async (person, category) => { + const read = await storage.personalKeys.get(person, category); + if (read !== null) await storage.personalKeys.remove(person, read.fingerprint); return read; }, }; @@ -330,9 +338,9 @@ describe("withPersonalData", () => { const { storage, store, registered, placed } = await setup(); const restricting: PersonalKeyStore = { ...storage.personalKeys, - get: async (person) => { + get: async (person, category) => { await storage.persons.restrict(person, new Date()); - return storage.personalKeys.get(person); + return storage.personalKeys.get(person, category); }, }; const refused = await withPersonalData({ @@ -387,7 +395,7 @@ describe("withPersonalData", () => { persons: [read], touches: ["c-4"], }); - const key = await storage.personalKeys.get("c-2"); + const key = await storage.personalKeys.get("c-2", DEFAULT_PERSONAL_CATEGORY); expect(appended[0]?.persons).toEqual([read, { person: "c-2", fingerprint: key?.fingerprint }]); expect(appended[0]?.touches).toEqual(["c-4", "c-2"]); await recording.append({ @@ -447,7 +455,7 @@ describe("withPersonalData", () => { }), ], }); - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); const read = [{ person: "c-1", fingerprint: key?.fingerprint }]; const [registered, renamed] = await store.readAll({ afterPosition: 0, limit: 10 }); expect(registered?.payload).toEqual({ email: "ana@example.com", plan: "free" }); @@ -459,7 +467,7 @@ describe("withPersonalData", () => { eventId: registered?.id ?? "", }); expect(openedUnder(one as StoredEvent)).toEqual(read); - await storage.personalKeys.remove("c-1"); + await removeKeys(storage.personalKeys, "c-1"); const [forgotten] = await store.readAll({ afterPosition: 0, limit: 1 }); expect(openedUnder(forgotten as StoredEvent)).toEqual([]); }); @@ -467,18 +475,8 @@ describe("withPersonalData", () => { it("keeps the key another writer added first", async () => { const { storage, store, registered } = await setup(); const first = { key: "Zmlyc3Q=", fingerprint: "first", kid: null }; - await storage.personalKeys.add("c-1", first); - const concurrentKeys: PersonalKeyStore = { - get: async () => null, - add: (person, key) => storage.personalKeys.add(person, key), - remove: (person) => storage.personalKeys.remove(person), - listWrappedUnder: (args) => storage.personalKeys.listWrappedUnder(args), - rewrap: (person, from, to) => storage.personalKeys.rewrap(person, from, to), - hold: (person, from, hold) => storage.personalKeys.hold(person, from, hold), - nextHoldEnd: (args) => storage.personalKeys.nextHoldEnd(args), - countWrappedUnder: (args) => storage.personalKeys.countWrappedUnder(args), - releaseHolds: () => storage.personalKeys.releaseHolds(), - }; + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, first); + const concurrentKeys: PersonalKeyStore = { ...storage.personalKeys, get: async () => null }; await expect( withPersonalData({ eventStore: storage.eventStore, @@ -492,7 +490,7 @@ describe("withPersonalData", () => { events: [registered(1, "ana@example.com")], }), ).rejects.toThrow(); - expect(await storage.personalKeys.get("c-1")).toEqual(first); + expect(await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toEqual(first); void store; }); @@ -582,7 +580,7 @@ describe("withPersonalData", () => { [1, { email: "ana@example.com", plan: "pro" }], [2, { customerId: "c-1", address: { street: "x" }, total: 1 }], ]); - await storage.personalKeys.remove("c-1"); + await removeKeys(storage.personalKeys, "c-1"); const after = await store.readByPerson({ person: "c-1", afterPosition: 1, limit: 10 }); expect(after.map((event) => event.payload)).toEqual([ { customerId: "c-1", address: null, total: 1 }, @@ -665,14 +663,103 @@ describe("withPersonalData", () => { expectedVersion: 0, events: [placed({ customerId: 42, address: { street: "x" }, total: 1 })], }); - expect((await storage.personalKeys.get("42"))?.fingerprint).toHaveLength(24); + expect( + (await storage.personalKeys.get("42", DEFAULT_PERSONAL_CATEGORY))?.fingerprint, + ).toHaveLength(24); + }); + + it("seals the fields of each category under the person's key of that category, and forgets one category alone", async () => { + const { storage, store } = await setup(); + const billed = { name: "Ana", iban: "ES00", email: "ana@example.com" }; + await store.append({ + aggregateType: "customer", + aggregateId: "c-1", + expectedVersion: 0, + events: [ + pendingEvent({ + aggregateType: "customer", + aggregateId: "c-1", + version: 1, + type: "CustomerBilled", + payload: billed, + }), + ], + }); + const [stored] = await storage.eventStore.readAll({ afterPosition: 0, limit: 10 }); + const sealed = stored?.payload as Record; + const keyOf = async (category: string) => + (await storage.personalKeys.get("c-1", category))?.fingerprint; + const [profile, invoices, plain] = await Promise.all( + ["profile", "invoices", DEFAULT_PERSONAL_CATEGORY].map(keyOf), + ); + expect(new Set([profile, invoices, plain]).size).toBe(3); + expect(sealed.name?.$personal).toMatchObject({ category: "profile", key: profile }); + expect(sealed.iban?.$personal).toMatchObject({ category: "invoices", key: invoices }); + expect(sealed.email?.$personal).toMatchObject({ key: plain }); + expect(sealed.email?.$personal).not.toHaveProperty("category"); + + const loaded = await store.load({ aggregateType: "customer", aggregateId: "c-1" }); + expect(loaded.events[0]?.payload).toEqual(billed); + expect(loaded.opened).toHaveLength(3); + expect(openedUnder(loaded.events[0] as StoredEvent)).toEqual( + expect.arrayContaining([{ person: "c-1", fingerprint: invoices }]), + ); + + await storage.personalKeys.remove("c-1", invoices ?? ""); + const after = await store.load({ aggregateType: "customer", aggregateId: "c-1" }); + expect(after.events[0]?.payload).toEqual({ ...billed, iban: null }); + expect(after.opened?.map(({ fingerprint }) => fingerprint)).not.toContain(invoices); + }); + + it("opens what a key kept past a forgetting sealed, naming it retained, which the store checks as such", async () => { + const { storage, store, registered } = await setup(); + await store.append({ + aggregateType: "customer", + aggregateId: "c-1", + expectedVersion: 0, + events: [registered(1, "ana@example.com")], + }); + const fingerprint = + (await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY))?.fingerprint ?? ""; + await storage.personalKeys.retain("c-1", fingerprint, { + since: "2026-01-01T00:00:00.000Z", + lastActiveAt: "2025-12-01T00:00:00.000Z", + }); + const loaded = await store.load({ aggregateType: "customer", aggregateId: "c-1" }); + expect(loaded.events[0]?.payload).toEqual({ email: "ana@example.com", plan: "pro" }); + expect(loaded.opened).toEqual([{ person: "c-1", fingerprint, retained: true }]); + const append = (persons: readonly KeyedPerson[]) => + store.append({ + aggregateType: "customer", + aggregateId: "c-1", + expectedVersion: 1, + events: [registered(2, "ana@again.example")], + persons, + }); + await expect(append([{ person: "c-1", fingerprint }])).rejects.toBeInstanceOf( + ForgottenSinceLoadError, + ); + await append(loaded.opened ?? []); + const again = await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); + expect(again?.fingerprint).not.toBe(fingerprint); + expect( + (await store.load({ aggregateType: "customer", aggregateId: "c-1" })).events.map( + (event) => (event.payload as { email: unknown }).email, + ), + ).toEqual(["ana@example.com", "ana@again.example"]); }); }); describe("withPersonalData on a process's lifecycle events", () => { const processes = { byName: { - "signup.onboarding": { personal: { person: "customerId", fields: ["email"] } }, + "signup.onboarding": { + personal: { + person: "customerId", + fields: ["email"], + categories: { email: DEFAULT_PERSONAL_CATEGORY }, + }, + }, "signup.plain": { personal: null }, }, } as unknown as Pick; @@ -715,15 +802,16 @@ describe("withPersonalData on a process's lifecycle events", () => { expect(stored?.person).toBe("c-1"); expect(JSON.stringify(stored)).not.toContain("ana@example.com"); - const fingerprint = (await storage.personalKeys.get("c-1"))?.fingerprint; + const fingerprint = (await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)) + ?.fingerprint; const loaded = await store.load(stream); expect(loaded.events[0]?.payload).toEqual({ state, eventId: "e-1" }); - expect(loaded.fingerprints).toEqual(new Map([["c-1", fingerprint]])); + expect(loaded.opened).toEqual([{ person: "c-1", fingerprint }]); expect(openedUnder(loaded.events[0] as StoredEvent)).toEqual([{ person: "c-1", fingerprint }]); const [byPerson] = await store.readByPerson({ person: "c-1", afterPosition: 0, limit: 10 }); expect(byPerson?.payload).toEqual({ state, eventId: "e-1" }); - await storage.personalKeys.remove("c-1"); + await removeKeys(storage.personalKeys, "c-1"); expect((await store.load(stream)).events[0]?.payload).toEqual({ state: { ...state, email: null }, eventId: "e-1", @@ -750,7 +838,7 @@ describe("withPersonalData on a process's lifecycle events", () => { [null, { state: { email: "a@b.c" } }], [null, { state: { email: "a@b.c" } }], ]); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); expect(await store.readAll({ afterPosition: 0, limit: 10 })).toEqual(stored); }); @@ -789,8 +877,9 @@ describe("withPersonalData on a process's lifecycle events", () => { const { storage, lifecycle, append } = await lifecycleSetup(); const state = { customerId: "c-1", email: "ana@example.com" }; await append([lifecycle(1, "ProcessHandled", { state })]); - const fingerprint = (await storage.personalKeys.get("c-1"))?.fingerprint ?? null; - await storage.personalKeys.remove("c-1"); + const fingerprint = + (await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY))?.fingerprint ?? null; + await removeKeys(storage.personalKeys, "c-1"); await expect( append([lifecycle(2, "ProcessHandled", { state })], { diff --git a/packages/core/src/kernel/personal-data/event-store.ts b/packages/core/src/kernel/personal-data/event-store.ts index aeb78ff0..03fe82da 100644 --- a/packages/core/src/kernel/personal-data/event-store.ts +++ b/packages/core/src/kernel/personal-data/event-store.ts @@ -16,7 +16,7 @@ import { seal, unseal, } from "./crypto.ts"; -import type { PersonalMarks } from "./marks.ts"; +import { fieldCategory, type PersonalMarks } from "./marks.ts"; import { lifecycleStateOf, sealLifecycleEvent } from "./process-state.ts"; import type { PersonalKeys } from "./wrapped-keys.ts"; @@ -78,15 +78,25 @@ export const carryOpening: CarryOpeningFunction = (from, to) => { */ const keyCache = () => { const keys = new Map>(); - return (person: string, open: () => Promise): Promise => { - const cached = keys.get(person); + return (person: string, under: string, open: () => Promise): Promise => { + const id = JSON.stringify([person, under]); + const cached = keys.get(id); if (cached !== undefined) return cached; const opened = open(); - keys.set(person, opened); + keys.set(id, opened); return opened; }; }; +// What a field is sealed or opened with: the person's key of a category, or of a fingerprint. +type SealingKey = (person: string, category: string) => Promise; +type OpeningKey = (person: string, fingerprint: string) => Promise; + +const keyedOf = (person: string, key: PersonalKey): KeyedPerson => + key.retained + ? { person, fingerprint: key.fingerprint, retained: true } + : { person, fingerprint: key.fingerprint }; + /** * The event store handlers write to and read from: personal fields go in encrypted with the key * of the person the event names, created on their first event, and come out decrypted, or as @@ -102,25 +112,30 @@ export const withPersonalData: WithPersonalDataFunction = ({ const marksOf = (event: Pick): PersonalMarks | null => aggregates.byName[event.aggregateType]?.eventsByType[event.type]?.personal ?? null; - // A person the state held under a key is sealed under that key alone: without it now, they were - // forgotten after the load, and any other key would make their old data readable again. - const keyInForce = async (args: AppendArgs, person: string): Promise => { - const stored = await keys.get(person); - const readUnder = (args.persons ?? []).filter( - (held) => held.person === person && held.fingerprint !== null, - ); - if (readUnder.some((held) => held.fingerprint !== stored?.fingerprint)) { - throw new ForgottenSinceLoadError(args, person); + // A person the state held under a key that is gone, or kept since where it was read in force, + // was forgotten after the load: no key is created for what was decided on their old data, which + // an append outside a transaction would leave behind. + const checkReads = async (args: AppendArgs, person: string): Promise => { + for (const { fingerprint, retained } of (args.persons ?? []).filter( + (held) => held.person === person, + )) { + if (fingerprint === null) continue; + const found = await keys.find(person, fingerprint); + if (found === null || (found.retainedSince !== null && retained !== true)) { + throw new ForgottenSinceLoadError(args, person); + } } - return stored === null ? keys.create(person) : keys.open(person, stored); }; + const keyInForce = async (person: string, category: string): Promise => + (await keys.held(person, category)) ?? keys.create(person, category); + // A lifecycle event names the person its state names, as a domain event does: the append is // refused while they are restricted, and their export finds it. const sealedState = async ( event: PendingEvent, process: string, - key: (person: string) => Promise, + key: SealingKey, touched: Set, ): Promise => { const state = lifecycleStateOf(event.payload); @@ -141,7 +156,7 @@ export const withPersonalData: WithPersonalDataFunction = ({ // `person`, and the append is refused while they are restricted. const sealed = async ( event: PendingEvent, - key: (person: string) => Promise, + key: SealingKey, touched: Set, ): Promise => { const process = processOfStream(event.aggregateType); @@ -161,7 +176,8 @@ export const withPersonalData: WithPersonalDataFunction = ({ for (const field of marks.fields) { const value = payload[field]; if (value === undefined) continue; - payload[field] = await seal({ key: await key(person), person, value }); + const category = fieldCategory(marks, field); + payload[field] = await seal({ key: await key(person, category), person, category, value }); } return { ...event, payload, person }; }; @@ -169,17 +185,20 @@ export const withPersonalData: WithPersonalDataFunction = ({ // `read` gets each person whose fields opened, with the key that opened them. const openFields = async ( fields: Readonly>, - key: (person: string) => Promise, - read: Map, + key: OpeningKey, + read: Map, ): Promise> => { const unsealed: Record = { ...fields }; for (const field in fields) { const value = fields[field]; if (!isSealed(value)) continue; const { person, key: sealedUnder } = (value as SealedValue).$personal; - const personal = await key(person); + const personal = await key(person, sealedUnder); unsealed[field] = await unseal({ key: personal, sealed: value as SealedValue }); - if (personal?.fingerprint === sealedUnder) read.set(person, sealedUnder); + if (personal !== null) { + const held = keyedOf(person, personal); + read.set(keyedPersonId(held), held); + } } return unsealed; }; @@ -198,20 +217,17 @@ export const withPersonalData: WithPersonalDataFunction = ({ event: StoredEvent, payload: Record, where: "payload" | "state", - key: (person: string) => Promise, - opened: Map, + key: OpeningKey, + opened: Map, ): Promise => { - const read = new Map(); + const read = new Map(); const unsealed = where === "state" ? { ...payload, state: await openFields(lifecycleStateOf(payload) ?? {}, key, read) } : await openFields(payload, key, read); - for (const [person, fingerprint] of read) opened.set(person, fingerprint); + for (const [id, held] of read) opened.set(id, held); const result = { ...event, payload: unsealed }; - openings.set( - result, - [...read].map(([person, fingerprint]) => ({ person, fingerprint })), - ); + openings.set(result, [...read.values()]); return result; }; @@ -219,8 +235,8 @@ export const withPersonalData: WithPersonalDataFunction = ({ // event comes back as it is, with no promise and no copy. const unsealed = ( event: StoredEvent, - key: (person: string) => Promise, - opened: Map, + key: OpeningKey, + opened: Map, ): StoredEvent | Promise => { const where = sealedIn(event); return where === null @@ -233,12 +249,10 @@ export const withPersonalData: WithPersonalDataFunction = ({ ? Promise.all(events) : Promise.resolve(events as StoredEvent[]); - const opening = (): ((person: string) => Promise) => { + const opening = (): OpeningKey => { const cache = keyCache(); - return (person) => - cache(person, async () => { - return keys.held(person); - }); + return (person, fingerprint) => + cache(person, fingerprint, () => keys.opened(person, fingerprint)); }; return { @@ -250,8 +264,11 @@ export const withPersonalData: WithPersonalDataFunction = ({ const persons = new Map(); for (const held of args.persons ?? []) persons.set(keyedPersonId(held), held); const touched = new Set(args.touches); - const key = async (person: string): Promise => { - const personal = await cache(person, () => keyInForce(args, person)); + const checked = new Map>(); + const key: SealingKey = async (person, category) => { + if (!checked.has(person)) checked.set(person, checkReads(args, person)); + await checked.get(person); + const personal = await cache(person, category, () => keyInForce(person, category)); const held = { person, fingerprint: personal.fingerprint }; persons.set(keyedPersonId(held), held); return personal; @@ -274,9 +291,9 @@ export const withPersonalData: WithPersonalDataFunction = ({ load: async (args) => { const loaded = await eventStore.load(args); const key = opening(); - const fingerprints = new Map(); - const events = await settled(loaded.events.map((e) => unsealed(e, key, fingerprints))); - return { ...loaded, events, fingerprints }; + const opened = new Map(); + const events = await settled(loaded.events.map((e) => unsealed(e, key, opened))); + return { ...loaded, events, opened: [...opened.values()] }; }, readOne: async (args) => { const event = await eventStore.readOne(args); @@ -284,12 +301,12 @@ export const withPersonalData: WithPersonalDataFunction = ({ }, readAll: async (args) => { const key = opening(); - const ignored = new Map(); + const ignored = new Map(); return settled((await eventStore.readAll(args)).map((e) => unsealed(e, key, ignored))); }, readByPerson: async (args) => { const key = opening(); - const ignored = new Map(); + const ignored = new Map(); return settled((await eventStore.readByPerson(args)).map((e) => unsealed(e, key, ignored))); }, lastPosition: () => eventStore.lastPosition(), diff --git a/packages/core/src/kernel/personal-data/export.test.ts b/packages/core/src/kernel/personal-data/export.test.ts index 64a993c4..f58368ba 100644 --- a/packages/core/src/kernel/personal-data/export.test.ts +++ b/packages/core/src/kernel/personal-data/export.test.ts @@ -13,6 +13,7 @@ import { buildAggregates } from "../aggregate/build-aggregates.ts"; import { type AppStorage, appStorage } from "../events/app-event-store.ts"; import { buildReadModels } from "../read-model/build-read-models.ts"; import { keysOver } from "../test-support.ts"; +import { buildPersonalCategories } from "./categories.ts"; import { sealCommandPayload } from "./commands.ts"; import { sealError } from "./errors.ts"; import { exportPersonalData } from "./export.ts"; @@ -20,6 +21,9 @@ import { createPersonalKeyWrapping } from "./wrapped-keys.ts"; type Table = RowTable>; +const DAY = 86_400_000; +const invoices = { category: "invoices" as never }; + const registry: Registry = { aggregates: { customer: { @@ -49,7 +53,7 @@ const registry: Registry = { payload: ({ z, personal, person }: EventPayloadArgs) => z.object({ customerId: person(z.string()), - note: personal(z.string()).optional(), + note: personal(z.string(), invoices).optional(), total: z.number(), }), evolve: () => ({}), @@ -77,7 +81,7 @@ const registry: Registry = { fields: ({ f }: FieldsArgs) => ({ orderId: f.string().primaryKey(), customerId: f.string().person(), - note: f.string().personal().optional(), + note: f.string().personal(invoices).optional(), total: f.number(), }), }, @@ -108,6 +112,17 @@ const registry: Registry = { queries: {}, }, }, + personalData: { categories: { invoices: { basis: "legal-obligation", retention: "60d" } } }, +}; + +const categories = buildPersonalCategories({ + module: registry.personalData, + retention: { months: 0, ms: 30 * DAY }, +}); + +const exportedCategories = { + default: { basis: null, retention: "30d", restrictedUntil: null }, + invoices: { basis: "legal-obligation", retention: "60d", restrictedUntil: null }, }; const setup = async () => { @@ -138,6 +153,7 @@ const setup = async () => { readModels, clock: createFixedClock(), batchSize, + categories, personId, }); const register = (aggregateId: string, email: string) => @@ -251,6 +267,7 @@ describe("exportPersonalData", () => { expect(exported).toStrictEqual({ personId: "c-1", exportedAt: "2026-01-01T00:00:00.000Z", + categories: exportedCategories, events: [ { id: "customer-c-1-1", @@ -261,6 +278,7 @@ describe("exportPersonalData", () => { position: 1, timestamp: "2026-01-01T00:00:00.000Z", data: { email: "ana@example.com" }, + categories: { email: "default" }, }, { id: "order-o-1-1", @@ -271,6 +289,7 @@ describe("exportPersonalData", () => { position: 2, timestamp: "2026-01-01T00:00:00.000Z", data: { note: "ring twice" }, + categories: { note: "invoices" }, }, { id: "order-o-2-1", @@ -281,6 +300,7 @@ describe("exportPersonalData", () => { position: 4, timestamp: "2026-01-01T00:00:00.000Z", data: {}, + categories: {}, }, { id: "order-o-3-1", @@ -291,6 +311,7 @@ describe("exportPersonalData", () => { position: 6, timestamp: "2026-01-01T00:00:00.000Z", data: { note: "leave at door" }, + categories: { note: "invoices" }, }, { id: "customer-c-1-3", @@ -301,6 +322,7 @@ describe("exportPersonalData", () => { position: 7, timestamp: "2026-01-01T00:00:00.000Z", data: {}, + categories: {}, }, ], scheduledCommands: [], @@ -400,6 +422,7 @@ describe("exportPersonalData", () => { type: "RemindCustomer", executeAt: "2026-01-02T00:00:00.000Z", data: { email: "ana@example.com" }, + categories: { email: "default" }, }, ]); expect(exported.deadLetters).toStrictEqual([ @@ -410,8 +433,9 @@ describe("exportPersonalData", () => { errorMessage: "no mailbox for ana@example.com", errorStack: "Error: no mailbox for ana@example.com\n at handler", data: { email: "ana@example.com" }, + categories: { email: "default" }, }, - { id: "dl-3", kind: "scheduled", handler: "NudgeCustomer", data: {} }, + { id: "dl-3", kind: "scheduled", handler: "NudgeCustomer", data: {}, categories: {} }, { id: "dl-4", kind: "policy", @@ -419,6 +443,7 @@ describe("exportPersonalData", () => { errorMessage: "ana@example.com bounced", errorStack: "Error: ana@example.com bounced\n at handler", data: {}, + categories: {}, }, ]); expect(JSON.stringify(exported)).not.toContain("bo@example.com"); @@ -454,16 +479,18 @@ describe("exportPersonalData", () => { const exported = await exportOf("c-1"); expect(exported.readModels).toStrictEqual({ - customerDirectory: [{ key: "c-1", data: { email: "ana@example.com" } }], + customerDirectory: [ + { key: "c-1", data: { email: "ana@example.com" }, categories: { email: "default" } }, + ], orderNotes: [ - { key: "o-1", data: { note: "ring twice" } }, - { key: "o-3", data: { note: "leave at door" } }, - { key: "o-4", data: {} }, + { key: "o-1", data: { note: "ring twice" }, categories: { note: "invoices" } }, + { key: "o-3", data: { note: "leave at door" }, categories: { note: "invoices" } }, + { key: "o-4", data: {}, categories: {} }, ], orderCustomers: [ - { key: "o-1", data: {} }, - { key: "o-3", data: {} }, - { key: "o-4", data: {} }, + { key: "o-1", data: {}, categories: {} }, + { key: "o-3", data: {}, categories: {} }, + { key: "o-4", data: {}, categories: {} }, ], }); expect(await exportOf("c-1", 100)).toEqual(exported); @@ -510,6 +537,7 @@ describe("exportPersonalData", () => { readModels: { ...readModels, byName: { ...readModels.byName, orderNotes: writing } }, clock: createFixedClock(), batchSize: 3, + categories, personId: "c-1", }); expect(exported.readModels.orderNotes?.map((row) => row.key)).toEqual([ @@ -551,6 +579,7 @@ describe("exportPersonalData", () => { readModels: { ...readModels, byName: failing }, clock: createFixedClock(), batchSize: 2, + categories, personId: "c-1", }), ).rejects.toBe(down); @@ -573,11 +602,20 @@ describe("exportPersonalData", () => { ...withoutStack, person: "c-1", }); - await storage.personalKeys.remove("c-1"); + for (const { key } of await storage.personalKeys.list("c-1")) { + await storage.personalKeys.remove("c-1", key.fingerprint); + } const exported = await exportOf("c-1"); expect(exported.events.map((event) => event.data)).toEqual([{ email: null }]); expect(exported.deadLetters).toStrictEqual([ - { id: "dl-1", kind: "policy", handler: "order.p", errorMessage: null, data: {} }, + { + id: "dl-1", + kind: "policy", + handler: "order.p", + errorMessage: null, + data: {}, + categories: {}, + }, ]); expect(exported.scheduledCommands).toEqual([ { @@ -585,8 +623,47 @@ describe("exportPersonalData", () => { type: "RemindCustomer", executeAt: "2026-01-02T00:00:00.000Z", data: { email: null }, + categories: { email: "default" }, }, ]); expect(JSON.stringify(exported)).not.toContain("ana@example.com"); }); + + it("tells until when what a forgetting kept of a category under a legal obligation stays restricted", async () => { + const { exportOf, place, storage } = await setup(); + await place("o-1", { customerId: "c-1", note: "ring twice", total: 10 }); + const [invoice] = (await storage.personalKeys.list("c-1")).filter( + ({ category }) => category === "invoices", + ); + await storage.personalKeys.retain("c-1", invoice?.key.fingerprint ?? "", { + since: "2026-01-10T00:00:00.000Z", + lastActiveAt: "2026-01-05T00:00:00.000Z", + }); + const exported = await exportOf("c-1"); + expect(exported.categories).toEqual({ + ...exportedCategories, + invoices: { ...exportedCategories.invoices, restrictedUntil: "2026-03-06T00:00:00.000Z" }, + }); + expect(exported.events.map(({ data, categories }) => [data, categories])).toEqual([ + [{ note: "ring twice" }, { note: "invoices" }], + ]); + }); + + it("names the default category without a basis, and without a retention when the app sets none", async () => { + const { exportOf, storage, aggregates, readModels } = await setup(); + const exported = await exportPersonalData({ + storage, + aggregates, + readModels, + processes: { byName: {} }, + clock: createFixedClock(), + batchSize: 2, + categories: buildPersonalCategories({ module: undefined, retention: null }), + personId: "c-1", + }); + expect(exported.categories).toEqual({ + default: { basis: null, retention: null, restrictedUntil: null }, + }); + expect((await exportOf("c-1")).categories).toEqual(exportedCategories); + }); }); diff --git a/packages/core/src/kernel/personal-data/export.ts b/packages/core/src/kernel/personal-data/export.ts index caaa493c..54290f33 100644 --- a/packages/core/src/kernel/personal-data/export.ts +++ b/packages/core/src/kernel/personal-data/export.ts @@ -1,7 +1,11 @@ import type { ReadModelStorage } from "../../adapter/adapter.ts"; import type { DeadLetterKind, StoredDeadLetter } from "../../adapter/storage/dead-letter-store.ts"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import type { Clock } from "../../contracts/clock.ts"; +import { retentionEnd } from "../../contracts/duration.ts"; import type { StoredEvent } from "../../contracts/event.ts"; +import type { LegalBasis } from "../../modules/personal-data.ts"; +import type { FieldsRecord } from "../../modules/view.ts"; import type { AggregatesRuntime } from "../aggregate/runtime.ts"; import { scheduledCommandId } from "../command/pipeline.ts"; import type { AppStorage } from "../events/app-event-store.ts"; @@ -9,9 +13,10 @@ import type { ProcessesRuntime } from "../process/build-processes.ts"; import { processOfStream } from "../process/lifecycle.ts"; import type { ReadModelPersonalData, ReadModelsRuntime } from "../read-model/build-read-models.ts"; import { isRecord } from "../shared/merge-fields.ts"; +import { formatRetention, type PersonalCategories } from "./categories.ts"; import { openCommandPayload, sealedKeys } from "./commands.ts"; import { openTextWith } from "./errors.ts"; -import type { PersonalMarks } from "./marks.ts"; +import { fieldCategory, type PersonalMarks } from "./marks.ts"; import { lifecycleStateOf } from "./process-state.ts"; import { checkPersonId } from "./transact.ts"; @@ -21,6 +26,23 @@ import { checkPersonId } from "./transact.ts"; */ export type ExportedData = Readonly>; +/** + * The category of each field of `data`, by field, which the export's `categories` describes. + */ +export type ExportedCategories = Readonly>; + +/** + * A category of the app as the export tells the person of it (article 15(1) of the GDPR): why it + * is processed, `null` for the default category, which declares no basis, and for how long after + * their last activity it is kept, `null` when nothing forgets it for inactivity. `restrictedUntil` + * is when what a forgetting kept of it under a legal obligation goes, `null` when it kept nothing. + */ +export interface ExportedCategory { + readonly basis: LegalBasis | null; + readonly retention: string | null; + readonly restrictedUntil: string | null; +} + /** * An event that names the person, with the personal fields it holds of them and nothing else of * its payload: for a process's lifecycle event, those of the state it carries. @@ -34,6 +56,7 @@ export interface ExportedEvent { readonly position: number; readonly timestamp: string; readonly data: ExportedData; + readonly categories: ExportedCategories; } /** @@ -44,6 +67,7 @@ export interface ExportedScheduledCommand { readonly type: string; readonly executeAt: string; readonly data: ExportedData; + readonly categories: ExportedCategories; } /** @@ -68,6 +92,7 @@ export interface ExportedDeadLetter { */ readonly errorStack?: string | null; readonly data: ExportedData; + readonly categories: ExportedCategories; } /** @@ -89,6 +114,7 @@ export interface ExportedSubscriberFailure { export interface ExportedRow { readonly key: unknown; readonly data: ExportedData; + readonly categories: ExportedCategories; } /** @@ -103,6 +129,10 @@ export interface PersonalDataExport { * When the export was taken, on the app's clock. */ readonly exportedAt: string; + /** + * Every category of the app, the default one included, by name. + */ + readonly categories: Readonly>; readonly events: readonly ExportedEvent[]; readonly scheduledCommands: readonly ExportedScheduledCommand[]; readonly deadLetters: readonly ExportedDeadLetter[]; @@ -126,6 +156,7 @@ export interface ExportPersonalDataArgs { readonly readModels: ReadModelsRuntime; readonly clock: Clock; readonly batchSize: number; + readonly categories: PersonalCategories; readonly personId: string; } @@ -133,13 +164,18 @@ export interface ExportPersonalDataFunction { (args: ExportPersonalDataArgs): Promise; } -const markedFields = (marks: PersonalMarks | null, payload: unknown): ExportedData => { - if (marks === null || !isRecord(payload)) return {}; - return Object.fromEntries( - marks.fields - .filter((field) => payload[field] !== undefined) - .map((field) => [field, payload[field]]), - ); +interface Marked { + readonly data: ExportedData; + readonly categories: ExportedCategories; +} + +const markedFields = (marks: PersonalMarks | null, payload: unknown): Marked => { + if (marks === null || !isRecord(payload)) return { data: {}, categories: {} }; + const fields = marks.fields.filter((field) => payload[field] !== undefined); + return { + data: Object.fromEntries(fields.map((field) => [field, payload[field]])), + categories: Object.fromEntries(fields.map((field) => [field, fieldCategory(marks, field)])), + }; }; // Each read model through its own storage, which may be another adapter's. Paged by primary key @@ -148,6 +184,7 @@ const markedFields = (marks: PersonalMarks | null, payload: unknown): ExportedDa const exportedRows = async ( storage: ReadModelStorage, { person, personal, primaryKey }: ReadModelPersonalData, + fields: FieldsRecord, personId: string, batchSize: number, ): Promise => { @@ -164,10 +201,12 @@ const exportedRows = async ( limit: batchSize, }); for (const row of page) { + const held = personal.filter((field) => row[field] !== undefined); rows.push({ key: row[primaryKey], - data: Object.fromEntries( - personal.filter((field) => row[field] !== undefined).map((field) => [field, row[field]]), + data: Object.fromEntries(held.map((field) => [field, row[field]])), + categories: Object.fromEntries( + held.map((field) => [field, fields[field]?.category ?? DEFAULT_PERSONAL_CATEGORY]), ), }); } @@ -178,6 +217,36 @@ const exportedRows = async ( return rows; }; +// What a forgetting kept of a category goes once its retention has run from the activity it kept. +const categoriesOf = async ( + storage: AppStorage, + categories: PersonalCategories, + personId: string, +): Promise>> => { + const held = await storage.personalKeys.list(personId); + return Object.fromEntries( + categories.all.map(({ name, basis, retention }) => { + const kept = + retention === null + ? [] + : held + .filter( + ({ category, retainedSince, lastActiveAt }) => + category === name && retainedSince !== null && lastActiveAt !== null, + ) + .map(({ lastActiveAt }) => retentionEnd(Date.parse(lastActiveAt ?? ""), retention)); + return [ + name, + { + basis, + retention: retention === null ? null : formatRetention(retention), + restrictedUntil: kept.length === 0 ? null : new Date(Math.max(...kept)).toISOString(), + }, + ]; + }), + ); +}; + // Nothing is locked: the export holds what each store held as it was read. export const exportPersonalData: ExportPersonalDataFunction = async ({ storage, @@ -186,6 +255,7 @@ export const exportPersonalData: ExportPersonalDataFunction = async ({ readModels, clock, batchSize, + categories, personId, }) => { checkPersonId(personId, "exportPersonalData"); @@ -193,19 +263,20 @@ export const exportPersonalData: ExportPersonalDataFunction = async ({ // Independent of the store's records, and often on another adapter, so read meanwhile; a // failure of the store's reads first leaves this one handled, the export rejecting with that. const rows = Promise.all( - Object.values(readModels.byName).flatMap(({ name, storage: rowsStorage, personalData }) => - personalData === null - ? [] - : [ - exportedRows(rowsStorage, personalData, personId, batchSize).then( - (exported) => [name, exported] as const, - ), - ], + Object.values(readModels.byName).flatMap( + ({ name, storage: rowsStorage, personalData, fields }) => + personalData === null + ? [] + : [ + exportedRows(rowsStorage, personalData, fields, personId, batchSize).then( + (exported) => [name, exported] as const, + ), + ], ), ); rows.catch(() => undefined); // A process's lifecycle event holds the fields its state marks, in that state. - const eventData = (event: StoredEvent): ExportedData => { + const eventData = (event: StoredEvent): Marked => { const process = processOfStream(event.aggregateType); if (process === null) { const marks = aggregates.byName[event.aggregateType]?.eventsByType[event.type]?.personal; @@ -221,7 +292,7 @@ export const exportPersonalData: ExportPersonalDataFunction = async ({ // Each person's key read once for the whole export. const keyOf = sealedKeys(storage.personalKeys); const openText = openTextWith(keyOf); - const openedData = async (type: string, payload: unknown): Promise => { + const openedData = async (type: string, payload: unknown): Promise => { const opened = await openCommandPayload({ payload, keys: storage.personalKeys, keyOf }); return markedFields(commandMarks(type), opened.payload); }; @@ -237,7 +308,7 @@ export const exportPersonalData: ExportPersonalDataFunction = async ({ ...(theirs && letter.errorStack !== undefined ? { errorStack: await openText(letter.errorStack) } : {}), - data: await openedData(letter.eventType, letter.payload), + ...(await openedData(letter.eventType, letter.payload)), }; }; @@ -259,7 +330,7 @@ export const exportPersonalData: ExportPersonalDataFunction = async ({ version: event.version, position: event.position, timestamp: event.timestamp, - data: eventData(event), + ...eventData(event), }); afterPosition = event.position; } @@ -270,7 +341,7 @@ export const exportPersonalData: ExportPersonalDataFunction = async ({ commandId: scheduledCommandId(entry.dedupeKey) ?? entry.dedupeKey, type: entry.command.type, executeAt: entry.executeAt, - data: await openedData(entry.command.type, entry.command.payload), + ...(await openedData(entry.command.type, entry.command.payload)), })), ); const deadLetters = await Promise.all( @@ -293,6 +364,7 @@ export const exportPersonalData: ExportPersonalDataFunction = async ({ return { personId, exportedAt, + categories: await categoriesOf(storage, categories, personId), events, scheduledCommands, deadLetters, diff --git a/packages/core/src/kernel/personal-data/forget.test.ts b/packages/core/src/kernel/personal-data/forget.test.ts index 808d5436..608c15d0 100644 --- a/packages/core/src/kernel/personal-data/forget.test.ts +++ b/packages/core/src/kernel/personal-data/forget.test.ts @@ -14,17 +14,28 @@ import { createFixedClock } from "../../contracts/clock.ts"; import { ConcurrencyError, ValidationError } from "../../contracts/errors.ts"; import { createSequentialIdGenerator } from "../../contracts/ids.ts"; import { memory } from "../../memory/index.ts"; +import type { ForgetOptions, PersonalDataModule } from "../../modules/personal-data.ts"; import { createRecordingLogger, withoutLedger } from "../test-support.ts"; +import { buildPersonalCategories } from "./categories.ts"; import { forgetIntended, forgetPersonalData, forgetWithin, STALE_INTENTION } from "./forget.ts"; import { boundForgetLedger } from "./forget-ledger.ts"; import { createForgetIntentions } from "./intention.ts"; -const setup = async (storage?: Storage) => { +const DAY_MS = 86_400_000; + +const declared: PersonalDataModule = { + categories: { + profile: { basis: "consent", retention: "30d" }, + invoices: { basis: "legal-obligation", retention: "30d" }, + }, +}; + +const setup = async (storage?: Storage, module?: PersonalDataModule) => { const live = storage ?? (await memory().createStorage()); const { logger, entries: logs } = createRecordingLogger(); const ids = createSequentialIdGenerator(); const clock = createFixedClock(); - const forget = (personId: string) => + const forget = (personId: string, options?: ForgetOptions) => forgetPersonalData({ storage: live, intentions: withoutLedger({ ids, clock }), @@ -32,15 +43,26 @@ const setup = async (storage?: Storage) => { clock, logger, concurrencyRetries: 2, + categories: buildPersonalCategories({ module, retention: null }), + batchSize: 10, personId, + options, }); - return { storage: live, forget, logs }; + return { storage: live, forget, logs, clock }; +}; + +const forgottenFields = { + reason: "requested", + whole: true, + categories: ["default"], + retained: [], + expired: [], }; describe("forgetPersonalData", () => { it("removes the person's key and appends PersonalDataForgotten to their stream, answering like a command", async () => { const { storage, forget, logs } = await setup(); - await storage.personalKeys.add("c-1", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); expect(await forget("c-1")).toEqual({ scheduled: false, aggregateType: "bounda", @@ -50,7 +72,7 @@ describe("forgetPersonalData", () => { eventTypes: ["PersonalDataForgotten"], position: 1, }); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.list("c-1")).toEqual([]); expect(await storage.eventStore.readAll({ afterPosition: 0, limit: 10 })).toEqual([ { id: "id-2", @@ -61,7 +83,7 @@ describe("forgetPersonalData", () => { type: "PersonalDataForgotten", payload: { personId: "c-1", - reason: "requested", + ...forgottenFields, fingerprints: ["f-1"], cancelledCommands: [], removedDeadLetters: [], @@ -82,8 +104,8 @@ describe("forgetPersonalData", () => { message: "personal data forgotten", fields: { personId: "c-1", - reason: "requested", - hadKey: true, + ...forgottenFields, + destroyedKeys: 1, cancelledCommands: 0, removedDeadLetters: 0, }, @@ -98,15 +120,15 @@ describe("forgetPersonalData", () => { expect(logs.map((entry) => entry.fields)).toEqual([ { personId: "c-2", - reason: "requested", - hadKey: false, + ...forgottenFields, + destroyedKeys: 0, cancelledCommands: 0, removedDeadLetters: 0, }, { personId: "c-2", - reason: "requested", - hadKey: false, + ...forgottenFields, + destroyedKeys: 0, cancelledCommands: 0, removedDeadLetters: 0, }, @@ -138,7 +160,7 @@ describe("forgetPersonalData", () => { const [event] = await storage.eventStore.readAll({ afterPosition: 0, limit: 10 }); expect(event?.payload).toEqual({ personId: "c-1", - reason: "requested", + ...forgottenFields, fingerprints: [], cancelledCommands: [ { commandId: "cmd-1", type: "RemindCustomer" }, @@ -152,13 +174,173 @@ describe("forgetPersonalData", () => { expect((await storage.deadLetterStore.list()).map((letter) => letter.id)).toEqual(["dl-2"]); expect(logs.at(-1)?.fields).toEqual({ personId: "c-1", - reason: "requested", - hadKey: false, + ...forgottenFields, + destroyedKeys: 0, cancelledCommands: 2, removedDeadLetters: 1, }); }); + it("forgets only the categories named, keeping what is sealed under the other keys", async () => { + const { storage, forget, logs } = await setup(undefined, declared); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "profile", testPersonalKey("2")); + const schedule = (dedupeKey: string, named: readonly KeyedPerson[]) => + storage.scheduler.schedule({ + dedupeKey, + command: testCommand("c-1"), + executeAt: new Date("2026-01-01T00:01:00.000Z"), + context: testContext, + persons: named, + }); + await schedule("command:cmd-1", [testKeyedPerson("c-1", "1")]); + await schedule("command:cmd-2", [testKeyedPerson("c-1", "2")]); + await schedule("command:cmd-3", [testKeyedPerson("c-1", null)]); + await storage.deadLetterStore.add({ + ...testDeadLetter("dl-1"), + persons: [testKeyedPerson("c-1", "1")], + }); + await storage.deadLetterStore.add({ + ...testDeadLetter("dl-2"), + persons: [testKeyedPerson("c-1", "2")], + }); + await forget("c-1", { categories: ["profile"] }); + expect((await storage.personalKeys.list("c-1")).map(({ category }) => category)).toEqual([ + "default", + ]); + expect((await storage.scheduler.list()).map((entry) => entry.dedupeKey)).toEqual([ + "command:cmd-1", + ]); + expect((await storage.deadLetterStore.list()).map((letter) => letter.id)).toEqual(["dl-1"]); + const [event] = await storage.eventStore.readAll({ afterPosition: 0, limit: 10 }); + expect(event?.payload).toEqual({ + personId: "c-1", + reason: "requested", + whole: false, + categories: ["profile"], + retained: [], + expired: [], + fingerprints: ["f-2"], + cancelledCommands: [ + { commandId: "cmd-2", type: "RemindCustomer" }, + { commandId: "cmd-3", type: "RemindCustomer" }, + ], + removedDeadLetters: [{ id: "dl-2", commandType: "order.p" }], + }); + expect(logs.at(-1)?.fields).toMatchObject({ whole: false, destroyedKeys: 1 }); + }); + + it("retains a key under a legal obligation while its retention runs from the last activity", async () => { + const { storage, forget, clock } = await setup(undefined, declared); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("2")); + const lastActiveAt = new Date(clock.now().getTime() - 10 * DAY_MS).toISOString(); + await storage.persons.record({ person: "c-1", position: 1, lastActiveAt, forgottenAt: null }); + await forget("c-1"); + expect(await storage.personalKeys.list("c-1")).toEqual([ + { + category: "invoices", + key: testPersonalKey("2"), + retainedSince: clock.now().toISOString(), + lastActiveAt, + }, + ]); + const [event] = await storage.eventStore.readAll({ afterPosition: 0, limit: 10 }); + expect(event?.payload).toMatchObject({ + whole: true, + categories: ["default", "profile"], + retained: ["invoices"], + expired: [], + }); + }); + + it("retains a key under a legal obligation when its category is named, too", async () => { + const { storage, forget } = await setup(undefined, declared); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("2")); + await forget("c-1", { categories: ["invoices"] }); + expect( + (await storage.personalKeys.list("c-1")).map(({ retainedSince }) => retainedSince), + ).toEqual(["2026-01-01T00:00:00.000Z"]); + expect(await storage.personalKeys.get("c-1", "invoices")).toBeNull(); + const [event] = await storage.eventStore.readAll({ afterPosition: 0, limit: 10 }); + expect(event?.payload).toMatchObject({ + whole: false, + categories: [], + retained: ["invoices"], + expired: [], + }); + }); + + it("destroys a key under a legal obligation once its retention has run out", async () => { + const { storage, forget, clock } = await setup(undefined, declared); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("2")); + await storage.persons.record({ + person: "c-1", + position: 1, + lastActiveAt: new Date(clock.now().getTime() - 31 * DAY_MS).toISOString(), + forgottenAt: null, + }); + await forget("c-1"); + expect(await storage.personalKeys.list("c-1")).toEqual([]); + const [event] = await storage.eventStore.readAll({ afterPosition: 0, limit: 10 }); + expect(event?.payload).toMatchObject({ + whole: true, + categories: ["default", "profile", "invoices"], + retained: [], + expired: [], + }); + }); + + it("refuses categories that are empty or that the app does not have, forgetting nothing", async () => { + const { storage, forget } = await setup(undefined, declared); + await expect(forget("c-1", { categories: [] })).rejects.toThrow( + new ValidationError("forget needs at least one category when it names them", [ + { path: ["categories"], message: "Expected a non-empty array of categories" }, + ]), + ); + const refusal = await forget("c-1", { + categories: ["profile", "orders", 3, "bills"] as unknown as ["default"], + }).catch((error: unknown) => error); + expect(refusal).toEqual( + new ValidationError('forget names categories the app does not have: "orders", 3, "bills"', [ + { + path: ["categories"], + message: 'Expected categories app/personal-data.ts declares, or "default"', + }, + ]), + ); + expect(refusal).toMatchObject({ + issues: [ + { + path: ["categories"], + message: 'Expected categories app/personal-data.ts declares, or "default"', + }, + ], + }); + expect(await storage.eventStore.lastPosition()).toBe(0); + }); + + it("destroys a key under a legal obligation whose retention ends at the forgetting itself", async () => { + const { storage, forget, clock } = await setup(undefined, declared); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("2")); + await storage.persons.record({ + person: "c-1", + position: 1, + lastActiveAt: new Date(clock.now().getTime() - 30 * DAY_MS).toISOString(), + forgottenAt: null, + }); + await forget("c-1"); + expect(await storage.personalKeys.list("c-1")).toEqual([]); + }); + + it("destroys the key of a category the app no longer declares", async () => { + const { storage, forget } = await setup(undefined, declared); + await storage.personalKeys.add("c-1", "retired", testPersonalKey("9")); + await forget("c-1"); + expect(await storage.personalKeys.list("c-1")).toEqual([]); + }); + it("refuses an empty id", async () => { const { storage, forget } = await setup(); await expect(forget("")).rejects.toThrow( @@ -226,7 +408,7 @@ describe("forgetIntended", () => { it("takes a new intention that names a key given to the person between the intention and the commit", async () => { const base = await memory().createStorage(); - await base.personalKeys.add("c-1", testPersonalKey("1")); + await base.personalKeys.add("c-1", "default", testPersonalKey("1")); let raced = false; const racing: Storage = { ...base, @@ -234,8 +416,8 @@ describe("forgetIntended", () => { if (!raced) { raced = true; // Another forgetting committed meanwhile, and the person came back with a new key. - await base.personalKeys.remove("c-1"); - await base.personalKeys.add("c-1", testPersonalKey("2")); + await base.personalKeys.remove("c-1", "f-1"); + await base.personalKeys.add("c-1", "default", testPersonalKey("2")); } return base.transact(work); }, @@ -247,7 +429,10 @@ describe("forgetIntended", () => { ids, clock, concurrencyRetries: 0, + categories: buildPersonalCategories({ module: undefined, retention: null }), + batchSize: 10, personId: "c-1", + scope: { kind: "whole" }, reason: "requested", context: testContext, }); @@ -256,22 +441,22 @@ describe("forgetIntended", () => { ["id-2", ["f-2"]], ]); expect(forgotten).toMatchObject({ - hadKey: true, + destroyed: 1, event: { id: "id-2", payload: { fingerprints: ["f-2"] } }, }); - expect(await base.personalKeys.get("c-1")).toBeNull(); + expect(await base.personalKeys.get("c-1", "default")).toBeNull(); }); it("forgets a person whose key went since the intention, naming none", async () => { const base = await memory().createStorage(); - await base.personalKeys.add("c-1", testPersonalKey("1")); + await base.personalKeys.add("c-1", "default", testPersonalKey("1")); let raced = false; const racing: Storage = { ...base, transact: async (work) => { if (!raced) { raced = true; - await base.personalKeys.remove("c-1"); + await base.personalKeys.remove("c-1", "f-1"); } return base.transact(work); }, @@ -283,12 +468,15 @@ describe("forgetIntended", () => { ids, clock, concurrencyRetries: 0, + categories: buildPersonalCategories({ module: undefined, retention: null }), + batchSize: 10, personId: "c-1", + scope: { kind: "whole" }, reason: "requested", context: testContext, }); expect(recorded).toHaveLength(1); - expect(forgotten).toMatchObject({ hadKey: false, payload: { fingerprints: [] } }); + expect(forgotten).toMatchObject({ destroyed: 0, payload: { fingerprints: [] } }); }); it("answers with the forgetting a reconciliation committed under the intention's id meanwhile", async () => { @@ -323,6 +511,8 @@ describe("forgetIntended", () => { clock: createFixedClock(), logger: createRecordingLogger().logger, concurrencyRetries: 0, + categories: buildPersonalCategories({ module: undefined, retention: null }), + batchSize: 10, personId: "c-1", }); expect(dispatch).toMatchObject({ eventIds: ["id-1"], version: 1, position: 1 }); @@ -332,7 +522,7 @@ describe("forgetIntended", () => { it("reads the held event and checks the key only when its caller has not", async () => { const storage = await memory().createStorage(); - await storage.personalKeys.add("c-1", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); let reads = 0; const forget = (checked: boolean, fingerprints: readonly string[]) => storage.transact((transaction) => { @@ -358,15 +548,23 @@ describe("forgetIntended", () => { fingerprints, }, context: testContext, + categories: buildPersonalCategories({ module: undefined, retention: null }), + scope: { kind: "whole" }, + batchSize: 10, checked, }); }); expect(await forget(false, ["f-other"])).toBe(STALE_INTENTION); expect(reads).toBe(1); + // A caller that checked keeps a key the intention does not name: given after it. expect(await forget(true, ["f-other"])).toMatchObject({ - hadKey: true, - event: { id: "e-true" }, + destroyed: 0, + event: { + id: "e-true", + payload: { fingerprints: [], whole: false, categories: [], retained: [] }, + }, }); expect(reads).toBe(1); + expect(await storage.personalKeys.list("c-1")).toHaveLength(1); }); }); diff --git a/packages/core/src/kernel/personal-data/forget.ts b/packages/core/src/kernel/personal-data/forget.ts index 71b936c9..0d81b463 100644 --- a/packages/core/src/kernel/personal-data/forget.ts +++ b/packages/core/src/kernel/personal-data/forget.ts @@ -1,6 +1,8 @@ import type { Storage, StorageTransaction } from "../../adapter/adapter.ts"; import type { Clock } from "../../contracts/clock.ts"; import type { StoredDispatch } from "../../contracts/command.ts"; +import { type Retention, retentionEnd } from "../../contracts/duration.ts"; +import { ValidationError } from "../../contracts/errors.ts"; import type { StoredEvent } from "../../contracts/event.ts"; import type { IdGenerator } from "../../contracts/ids.ts"; import type { Logger } from "../../contracts/logger.ts"; @@ -11,12 +13,24 @@ import { type PersonalDataForgottenPayload, type PersonalDataForgottenReason, } from "../../modules/bounda-events.ts"; +import type { ForgetOptions, PersonalCategoryName } from "../../modules/personal-data.ts"; import { scheduledCommandId } from "../command/pipeline.ts"; import { actedBy, startedBy } from "../shared/causation.ts"; import { appendSystemEvent } from "../system-events.ts"; -import type { ForgetIntention, ForgetIntentions } from "./intention.ts"; +import { lastActivity } from "./activity.ts"; +import type { PersonalCategories } from "./categories.ts"; +import { coveredBy, type ForgetIntention, type ForgetIntentions } from "./intention.ts"; import { checkPersonId, transactRetrying } from "./transact.ts"; +/** + * What a forgetting covers: the person as a whole, some of their categories, or, for a category's + * retention, some of their keys by fingerprint. + */ +export type ForgetScope = + | { readonly kind: "whole" } + | { readonly kind: "categories"; readonly categories: readonly string[] } + | { readonly kind: "keys"; readonly fingerprints: readonly string[] }; + export interface ForgetWithinArgs { readonly transaction: StorageTransaction; readonly ids: IdGenerator; @@ -24,15 +38,24 @@ export interface ForgetWithinArgs { readonly personId: string; readonly intention: ForgetIntention; readonly context: CausationContext; + readonly categories: PersonalCategories; + readonly scope: ForgetScope; + /** + * How many events a read of the person's last activity takes at once. + */ + readonly batchSize: number; /** - * The caller found already, in this transaction, the intention's event not held and the - * person's key named by it. + * The caller found already, in this transaction, the intention's event not held and a key the + * scope covers named by it. */ readonly checked?: boolean; } export interface Forgotten { - readonly hadKey: boolean; + /** + * How many keys went for good. + */ + readonly destroyed: number; readonly payload: PersonalDataForgottenPayload; readonly event: StoredEvent; } @@ -45,6 +68,18 @@ export interface HeldForgetting { readonly held: StoredEvent; } +// A key in force of a category under a legal obligation outlives a request while its retention +// runs from the person's last activity, read past what the retention recorded so it is never cut +// short; a person never seen active is taken as active now. +const keptUntil = async ( + args: ForgetWithinArgs, + retention: Retention, + activity: () => Promise, +): Promise<{ readonly lastActiveAt: string; readonly until: number }> => { + const lastActiveAt = (await activity()) ?? args.clock.now().toISOString(); + return { lastActiveAt, until: retentionEnd(Date.parse(lastActiveAt), retention) }; +}; + /** * The person holds a key the intention does not name: given to them after it was written. */ @@ -57,53 +92,101 @@ export interface ForgetWithinFunction { } /** - * Removes the person's key and appends `PersonalDataForgotten` to their stream, with the id, time - * and reason of the intention, on the transaction given: whoever reads the event finds the fields - * gone already. The scheduled commands and dead letters that hold the person's data go in the same - * transaction, and the event names them. A person without a key is forgotten all the same, since - * read models may hold what their events carried in the clear. + * Forgets what `scope` covers on the transaction given, and appends `PersonalDataForgotten` to the + * person's stream, with the id, time and reason of the intention: whoever reads the event finds + * the fields gone already. A key in force of a category whose basis is a legal obligation is + * retained instead while its retention has not run out, when the person or the category is what + * was asked for; a key named by fingerprint goes whatever it is. The scheduled commands, dead + * letters, snapshots and remembered rejections that hold the person's data under a key no longer + * in force go in the same transaction, every one that names them when they are forgotten as a + * whole, and the event names the commands and the letters. A person without a key is forgotten + * all the same, since read models may hold what their events carried in the clear. */ -export const forgetWithin: ForgetWithinFunction = async ({ - transaction, - ids, - clock, - personId, - intention, - context, - checked = false, -}) => { +export const forgetWithin: ForgetWithinFunction = async (args) => { + const { transaction, ids, clock, personId, intention, context, categories, scope } = args; + const { checked = false } = args; const { fingerprints } = intention; if (fingerprints !== undefined && !checked) { - const held = await transaction.eventStore.readOne({ + const applied = await transaction.eventStore.readOne({ aggregateType: BOUNDA_AGGREGATE, aggregateId: personId, eventId: intention.eventId, }); - if (held !== null) return { held }; - } - const key = await transaction.personalKeys.get(personId); - if ( - !checked && - key !== null && - fingerprints !== undefined && - !fingerprints.includes(key.fingerprint) - ) { - return STALE_INTENTION; + if (applied !== null) return { held: applied }; } + const now = clock.now(); + // A key the scope covers that the intention does not name was given to the person after it: a + // caller that checked the intention already keeps it. + const covered = coveredBy(scope, await transaction.personalKeys.list(personId)); + const asked = covered.filter( + ({ key }) => fingerprints === undefined || fingerprints.includes(key.fingerprint), + ); + if (!checked && asked.length < covered.length) return STALE_INTENTION; + let activity: Promise | undefined; + const lastActive = (): Promise => { + activity ??= lastActivity({ + persons: transaction.persons, + eventStore: transaction.eventStore, + person: personId, + batchSize: args.batchSize, + }); + return activity; + }; // The key goes first: a snapshot written beside the transaction then either waits for it and - // is forgotten with the rest, or finds no key and is never written. The scheduled commands go - // before the dead letters: a worker giving one up writes its letter and settles its claim - // together, so either this waits for it and then finds the letter, or the claim is lost and the - // letter rolls back. - const hadKey = await transaction.personalKeys.remove(personId); - await transaction.snapshotStore.forget(personId); - await transaction.idempotencyStore.forget(personId); - const cancelled = await transaction.scheduler.forget(personId); - const letters = await transaction.deadLetterStore.forget(personId); + // is forgotten with the rest, or finds no key in force and is never written. The scheduled + // commands go before the dead letters: a worker giving one up writes its letter and settles its + // claim together, so either this waits for it and then finds the letter, or the claim is lost + // and the letter rolls back. + const gone = new Set(); + const expired = new Set(); + let destroyed = 0; + for (const { category, key, retainedSince } of asked) { + const basis = categories.of(category); + if (scope.kind !== "keys" && basis?.basis === "legal-obligation" && basis.retention !== null) { + const kept = await keptUntil(args, basis.retention, lastActive); + if (kept.until > now.getTime()) { + await transaction.personalKeys.retain(personId, key.fingerprint, { + since: now.toISOString(), + lastActiveAt: kept.lastActiveAt, + }); + continue; + } + } + if (await transaction.personalKeys.remove(personId, key.fingerprint)) destroyed += 1; + (retainedSince === null ? gone : expired).add(category); + } + // A forgetting as a whole keeps nothing, unless a caller that checked its intention left a key + // given after it in force; that key's category is then neither whole nor forgotten. + const after = await transaction.personalKeys.list(personId); + const inForce = after.filter(({ retainedSince }) => retainedSince === null); + const keep = inForce.map(({ key }) => key.fingerprint); + const inForceAfter = new Set(inForce.map(({ category }) => category)); + const retainedAfter = new Set( + after.filter(({ retainedSince }) => retainedSince !== null).map(({ category }) => category), + ); + await transaction.snapshotStore.forget(personId, keep); + await transaction.idempotencyStore.forget(personId, keep); + const cancelled = await transaction.scheduler.forget(personId, keep); + const letters = await transaction.deadLetterStore.forget(personId, keep); + const named: readonly string[] = + scope.kind === "whole" + ? categories.all.map(({ name }) => name) + : scope.kind === "categories" + ? scope.categories + : [...gone]; const payload: PersonalDataForgottenPayload = { personId, reason: intention.reason, - fingerprints: key === null ? [] : [key.fingerprint], + whole: scope.kind === "whole" && keep.length === 0, + categories: named.filter( + (category) => + gone.has(category) || (!retainedAfter.has(category) && !inForceAfter.has(category)), + ) as PersonalCategoryName[], + retained: (scope.kind === "keys" + ? [] + : named.filter((category) => retainedAfter.has(category))) as PersonalCategoryName[], + expired: [...expired] as PersonalCategoryName[], + fingerprints: asked.map(({ key }) => key.fingerprint), cancelledCommands: cancelled.map(({ dedupeKey, command }) => ({ commandId: scheduledCommandId(dedupeKey) ?? dedupeKey, type: command.type, @@ -122,7 +205,7 @@ export const forgetWithin: ForgetWithinFunction = async ({ person: personId, identity: { id: intention.eventId, timestamp: intention.forgottenAt }, }); - return { hadKey, payload, event }; + return { destroyed, payload, event }; }; export interface ForgetIntendedArgs { @@ -131,7 +214,10 @@ export interface ForgetIntendedArgs { readonly ids: IdGenerator; readonly clock: Clock; readonly concurrencyRetries: number; + readonly categories: PersonalCategories; + readonly batchSize: number; readonly personId: string; + readonly scope: ForgetScope; readonly reason: PersonalDataForgottenReason; readonly context: CausationContext; } @@ -151,17 +237,35 @@ export const forgetIntended: ForgetIntendedFunction = async ({ ids, clock, concurrencyRetries, + categories, + batchSize, personId, + scope, reason, context, }) => { for (;;) { - const intention = await intentions.intend({ personId, reason, keys: storage.personalKeys }); + const intention = await intentions.intend({ + personId, + reason, + scope, + keys: storage.personalKeys, + }); const outcome = await transactRetrying({ storage, concurrencyRetries, work: (transaction) => - forgetWithin({ transaction, ids, clock, personId, intention, context }), + forgetWithin({ + transaction, + ids, + clock, + personId, + intention, + context, + categories, + scope, + batchSize, + }), }); if (outcome !== STALE_INTENTION) return outcome; } @@ -171,11 +275,15 @@ export interface LogForgottenFunction { (logger: Logger, forgotten: Forgotten): void; } -export const logForgotten: LogForgottenFunction = (logger, { hadKey, payload, event }) => { +export const logForgotten: LogForgottenFunction = (logger, { destroyed, payload, event }) => { logger.info("personal data forgotten", { personId: payload.personId, reason: payload.reason, - hadKey, + whole: payload.whole, + categories: payload.categories, + retained: payload.retained, + expired: payload.expired, + destroyedKeys: destroyed, cancelledCommands: payload.cancelledCommands.length, removedDeadLetters: payload.removedDeadLetters.length, ...actedBy(event.metadata.actor), @@ -203,7 +311,10 @@ export interface ForgetPersonalDataArgs { readonly clock: Clock; readonly logger: Logger; readonly concurrencyRetries: number; + readonly categories: PersonalCategories; + readonly batchSize: number; readonly personId: string; + readonly options?: ForgetOptions | undefined; readonly actor?: string | undefined; } @@ -211,6 +322,44 @@ export interface ForgetPersonalDataFunction { (args: ForgetPersonalDataArgs): Promise; } +export interface ForgetScopeOfArgs { + readonly categories: PersonalCategories; + readonly options: ForgetOptions | undefined; +} + +export interface ForgetScopeOfFunction { + (args: ForgetScopeOfArgs): ForgetScope; +} + +/** + * The scope of `forget(personId, options)`: the person as a whole without `categories`. Throws + * `ValidationError` for categories that are empty or that the app does not have. + */ +export const forgetScopeOf: ForgetScopeOfFunction = ({ categories, options }) => { + if (options?.categories === undefined) return { kind: "whole" }; + const asked: unknown = options.categories; + if (!Array.isArray(asked) || asked.length === 0) { + throw new ValidationError("forget needs at least one category when it names them", [ + { path: ["categories"], message: "Expected a non-empty array of categories" }, + ]); + } + const unknown = asked.filter( + (category) => typeof category !== "string" || categories.of(category) === undefined, + ); + if (unknown.length > 0) { + throw new ValidationError( + `forget names categories the app does not have: ${unknown.map((category) => JSON.stringify(category)).join(", ")}`, + [ + { + path: ["categories"], + message: 'Expected categories app/personal-data.ts declares, or "default"', + }, + ], + ); + } + return { kind: "categories", categories: [...new Set(asked as string[])] }; +}; + /** * `forget` of a client: `forgetIntended`, as a request of the person. */ @@ -221,10 +370,14 @@ export const forgetPersonalData: ForgetPersonalDataFunction = async ({ clock, logger, concurrencyRetries, + categories, + batchSize, personId, + options, actor, }) => { checkPersonId(personId, "forget"); + const scope = forgetScopeOf({ categories, options }); const correlationId = ids.next(); const forgotten = await forgetIntended({ storage, @@ -232,7 +385,10 @@ export const forgetPersonalData: ForgetPersonalDataFunction = async ({ ids, clock, concurrencyRetries, + categories, + batchSize, personId, + scope, reason: "requested", context: startedBy(correlationId, actor), }); diff --git a/packages/core/src/kernel/personal-data/intention.test.ts b/packages/core/src/kernel/personal-data/intention.test.ts index 99e408b7..42ae95c5 100644 --- a/packages/core/src/kernel/personal-data/intention.test.ts +++ b/packages/core/src/kernel/personal-data/intention.test.ts @@ -6,11 +6,14 @@ import { ForgetLedgerUnavailableError } from "../../contracts/errors.ts"; import { createSequentialIdGenerator } from "../../contracts/ids.ts"; import { memory } from "../../memory/index.ts"; import { createRecordingLogger } from "../test-support.ts"; +import type { ForgetScope } from "./forget.ts"; import { type BoundedForgetLedger, boundForgetLedger } from "./forget-ledger.ts"; import { createForgetIntentions } from "./intention.ts"; const NOW = new Date("2026-03-01T00:00:00.000Z"); +const WHOLE: ForgetScope = { kind: "whole" }; + const setUp = async ({ ledger, strict = false, @@ -42,40 +45,68 @@ const recordingLedger = (append: (entries: readonly ForgetLedgerEntry[]) => Prom }); describe("the intentions of forgetting", () => { - it("record each in the ledger before they resolve, naming the key the person holds now", async () => { + it("record each in the ledger before they resolve, naming the keys in force their scope covers", async () => { const appended: ForgetLedgerEntry[] = []; const { storage, intentions } = await setUp({ ledger: recordingLedger(async (entries) => { appended.push(...entries); }), }); - await storage.personalKeys.add("c-1", testPersonalKey("1")); - expect( - await intentions.intend({ personId: "c-1", reason: "retention", keys: storage.personalKeys }), - ).toEqual({ + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "profile", testPersonalKey("2")); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("3")); + await storage.personalKeys.retain("c-1", "f-3", { + since: NOW.toISOString(), + lastActiveAt: NOW.toISOString(), + }); + const intend = (personId: string, scope: ForgetScope) => + intentions.intend({ personId, reason: "retention", scope, keys: storage.personalKeys }); + expect(await intend("c-1", WHOLE)).toEqual({ eventId: "id-1", forgottenAt: NOW.toISOString(), reason: "retention", - fingerprints: ["f-1"], + fingerprints: ["f-1", "f-2"], }); + expect(await intend("c-2", WHOLE)).toMatchObject({ eventId: "id-2", fingerprints: [] }); expect( - await intentions.intend({ personId: "c-2", reason: "requested", keys: storage.personalKeys }), - ).toMatchObject({ eventId: "id-2", fingerprints: [] }); + await intend("c-1", { kind: "categories", categories: ["profile", "invoices"] }), + ).toMatchObject({ fingerprints: ["f-2"] }); + expect(await intend("c-1", { kind: "keys", fingerprints: ["f-2", "f-3"] })).toMatchObject({ + fingerprints: ["f-2", "f-3"], + }); expect(appended).toEqual([ { personId: "c-1", + categories: null, eventId: "id-1", forgottenAt: NOW.toISOString(), reason: "retention", - fingerprints: ["f-1"], + fingerprints: ["f-1", "f-2"], }, { personId: "c-2", + categories: null, eventId: "id-2", forgottenAt: NOW.toISOString(), - reason: "requested", + reason: "retention", fingerprints: [], }, + { + personId: "c-1", + categories: ["profile", "invoices"], + eventId: "id-3", + forgottenAt: NOW.toISOString(), + reason: "retention", + fingerprints: ["f-2"], + }, + { + personId: "c-1", + categories: ["profile"], + eventId: "id-4", + forgottenAt: NOW.toISOString(), + reason: "retention", + fingerprints: ["f-2", "f-3"], + }, ]); }); @@ -83,18 +114,21 @@ describe("the intentions of forgetting", () => { const { storage, intentions } = await setUp({ ledger: null }); let reads = 0; const keys = { - get: async (person: string) => { + list: async (person: string) => { reads += 1; - return storage.personalKeys.get(person); + return storage.personalKeys.list(person); }, }; - expect(await intentions.intend({ personId: "c-1", reason: "requested", keys })).toEqual({ + expect( + await intentions.intend({ personId: "c-1", reason: "requested", scope: WHOLE, keys }), + ).toEqual({ eventId: "id-1", forgottenAt: NOW.toISOString(), reason: "requested", }); await intentions.record({ personId: "c-1", + categories: null, eventId: "e-1", forgottenAt: NOW.toISOString(), reason: "requested", @@ -113,6 +147,7 @@ describe("the intentions of forgetting", () => { const refused = intentions.intend({ personId: "c-1", reason: "requested", + scope: WHOLE, keys: storage.personalKeys, }); await expect(refused).rejects.toBeInstanceOf(ForgetLedgerUnavailableError); @@ -126,7 +161,12 @@ describe("the intentions of forgetting", () => { }), }); expect( - await intentions.intend({ personId: "c-1", reason: "requested", keys: storage.personalKeys }), + await intentions.intend({ + personId: "c-1", + reason: "requested", + scope: WHOLE, + keys: storage.personalKeys, + }), ).toMatchObject({ eventId: "id-1", fingerprints: [] }); expect(logs).toEqual([ { @@ -153,6 +193,7 @@ describe("the intentions of forgetting", () => { await loose.intentions.intend({ personId: "c-1", reason: "requested", + scope: WHOLE, keys: loose.storage.personalKeys, }); expect(appends).toBe(0); @@ -160,6 +201,7 @@ describe("the intentions of forgetting", () => { await strict.intentions.intend({ personId: "c-1", reason: "requested", + scope: WHOLE, keys: strict.storage.personalKeys, }); expect(appends).toBe(1); @@ -167,6 +209,7 @@ describe("the intentions of forgetting", () => { await loose.intentions.intend({ personId: "c-1", reason: "requested", + scope: WHOLE, keys: loose.storage.personalKeys, }); expect(appends).toBe(2); diff --git a/packages/core/src/kernel/personal-data/intention.ts b/packages/core/src/kernel/personal-data/intention.ts index e20ba631..daecc960 100644 --- a/packages/core/src/kernel/personal-data/intention.ts +++ b/packages/core/src/kernel/personal-data/intention.ts @@ -1,10 +1,14 @@ import type { ForgetLedgerEntry } from "../../adapter/storage/forget-ledger.ts"; -import type { PersonalKeyStore } from "../../adapter/storage/personal-key-store.ts"; +import type { + HeldPersonalKey, + PersonalKeyStore, +} from "../../adapter/storage/personal-key-store.ts"; import type { Clock } from "../../contracts/clock.ts"; import { ForgetLedgerUnavailableError } from "../../contracts/errors.ts"; import type { IdGenerator } from "../../contracts/ids.ts"; import type { Logger } from "../../contracts/logger.ts"; import type { PersonalDataForgottenReason } from "../../modules/bounda-events.ts"; +import type { ForgetScope } from "./forget.ts"; import type { BoundedForgetLedger } from "./forget-ledger.ts"; /** @@ -15,9 +19,10 @@ export interface ForgetIntention { readonly forgottenAt: string; readonly reason: PersonalDataForgottenReason; /** - * The keys the intention names, which the forgetting may destroy: one the person holds that is - * not here was given to them after the intention was written. `undefined` without a ledger, and - * for an entry that does not record them, which any key matches. + * The keys the intention covers, which the forgetting may destroy or retain: one in force the + * person holds in its scope that is not here was given to them after the intention was written. + * `undefined` without a ledger, and for an entry that does not record them, which any key + * matches. */ readonly fingerprints?: readonly string[] | undefined; } @@ -25,16 +30,17 @@ export interface ForgetIntention { export interface IntendArgs { readonly personId: string; readonly reason: PersonalDataForgottenReason; + readonly scope: ForgetScope; /** - * Where the key the person holds now is read, outside any transaction. + * Where the keys the person holds now are read, outside any transaction. */ - readonly keys: Pick; + readonly keys: Pick; } export interface ForgetIntentions { /** - * The intention of forgetting the person now, naming the key they hold, recorded in the ledger - * before it resolves. + * The intention of forgetting what `scope` covers of the person now, naming the keys it covers, + * recorded in the ledger before it resolves. */ intend(args: IntendArgs): Promise; /** @@ -45,6 +51,39 @@ export interface ForgetIntentions { record(entry: ForgetLedgerEntry): Promise; } +export interface CoveredByFunction { + (scope: ForgetScope, held: readonly HeldPersonalKey[]): readonly HeldPersonalKey[]; +} + +/** + * The keys a forgetting of `scope` covers among those the person holds: every one in force for the + * person as a whole, those in force of the categories named, or the keys named by fingerprint, + * whatever they are. + */ +export const coveredBy: CoveredByFunction = (scope, held) => + scope.kind === "keys" + ? held.filter(({ key }) => scope.fingerprints.includes(key.fingerprint)) + : held.filter( + ({ category, retainedSince }) => + retainedSince === null && (scope.kind === "whole" || scope.categories.includes(category)), + ); + +// What the ledger keeps a forgetting asked for: the categories of the keys in force it names by +// fingerprint, so a retention's comes back as the categories it ended, and none for the keys an +// earlier forgetting retained, which the retention ends again on its own. +const askedFor = ( + scope: ForgetScope, + covered: readonly HeldPersonalKey[], +): readonly string[] | null => { + if (scope.kind === "whole") return null; + if (scope.kind === "categories") return scope.categories; + return [ + ...new Set( + covered.filter(({ retainedSince }) => retainedSince === null).map(({ category }) => category), + ), + ]; +}; + export interface CreateForgetIntentionsArgs { /** * `null` without a ledger: nothing is recorded, and no key is read for it. @@ -91,13 +130,20 @@ export const createForgetIntentions: CreateForgetIntentionsFunction = ({ } }; return { - intend: async ({ personId, reason, keys }) => { + intend: async ({ personId, reason, scope, keys }) => { const eventId = ids.next(); const forgottenAt = clock.now().toISOString(); if (ledger === null) return { eventId, forgottenAt, reason }; - const key = await keys.get(personId); - const fingerprints = key === null ? [] : [key.fingerprint]; - await record({ personId, eventId, forgottenAt, reason, fingerprints }); + const covered = coveredBy(scope, await keys.list(personId)); + const fingerprints = covered.map(({ key }) => key.fingerprint); + await record({ + personId, + categories: askedFor(scope, covered), + eventId, + forgottenAt, + reason, + fingerprints, + }); return { eventId, forgottenAt, reason, fingerprints }; }, record, diff --git a/packages/core/src/kernel/personal-data/key-wrapping-app.test.ts b/packages/core/src/kernel/personal-data/key-wrapping-app.test.ts index 33a47368..4839a354 100644 --- a/packages/core/src/kernel/personal-data/key-wrapping-app.test.ts +++ b/packages/core/src/kernel/personal-data/key-wrapping-app.test.ts @@ -131,8 +131,11 @@ const emailOf = async (app: BoundaApp, customerId: string) => (await app.exportPersonalData(customerId)).events[0]?.data; const keyRows = (db: ReturnType["db"]) => - db.prepare(`SELECT "person", "key", "fingerprint", "kid" FROM "bounda_personal_keys"`).all() as { + db + .prepare(`SELECT "person", "category", "key", "fingerprint", "kid" FROM "bounda_personal_keys"`) + .all() as { person: string; + category: string; key: string; fingerprint: string; kid: string | null; @@ -197,8 +200,14 @@ describe("personal keys wrapped under a master key", () => { expect(await app.forgetAgain([{ personId: "c-1" }])).toEqual(["c-1"]); // The same row put back, as a restore would: unwrapped again, so the app had let it go. db.prepare( - `INSERT INTO "bounda_personal_keys" ("person", "key", "fingerprint", "kid") VALUES (?, ?, ?, ?)`, - ).run(row?.person ?? "", row?.key ?? "", row?.fingerprint ?? "", row?.kid ?? null); + `INSERT INTO "bounda_personal_keys" ("person", "category", "key", "fingerprint", "kid") VALUES (?, ?, ?, ?, ?)`, + ).run( + row?.person ?? "", + row?.category ?? "", + row?.key ?? "", + row?.fingerprint ?? "", + row?.kid ?? null, + ); const before = unwraps; await emailOf(app, "c-1"); expect(unwraps).toBe(before + 1); diff --git a/packages/core/src/kernel/personal-data/ledger-outbox.test.ts b/packages/core/src/kernel/personal-data/ledger-outbox.test.ts index f53b46a3..de5a2ca5 100644 --- a/packages/core/src/kernel/personal-data/ledger-outbox.test.ts +++ b/packages/core/src/kernel/personal-data/ledger-outbox.test.ts @@ -61,23 +61,41 @@ describe("the forget ledger's outbox", () => { version: 1, type: "PersonalDataForgotten", id: "f-2", + payload: { whole: false, categories: ["profile"], retained: ["invoices"], expired: [] }, + }), + stored(6, { + aggregateType: "bounda", + aggregateId: "c-3", + version: 1, + type: "PersonalDataForgotten", + id: "f-3", + payload: { whole: false, categories: [], retained: [], expired: ["invoices"] }, }), ]; - expect(await outbox.process(events, span)).toBe(5); + expect(await outbox.process(events, span)).toBe(6); expect(appended).toEqual([ [ { personId: "c-1", + categories: null, eventId: "f-1", forgottenAt: "2026-01-01T00:00:00.000Z", reason: "requested", }, { personId: "c-2", + categories: ["profile", "invoices"], eventId: "f-2", forgottenAt: "2026-01-01T00:00:00.000Z", reason: "requested", }, + { + personId: "c-3", + categories: [], + eventId: "f-3", + forgottenAt: "2026-01-01T00:00:00.000Z", + reason: "requested", + }, ], ]); expect(await outbox.process(events.slice(0, 3), span)).toBe(3); @@ -118,6 +136,7 @@ describe("the forget ledger's outbox", () => { [ { personId: "c-3", + categories: null, eventId: "f-3", forgottenAt: "2026-01-01T00:01:00.001Z", reason: "requested", @@ -146,6 +165,7 @@ describe("a forgetting's entry", () => { forgettingEntry(forgetting({ personId: "c-1", reason: "retention", fingerprints: ["k-1"] })), ).toEqual({ personId: "c-1", + categories: null, eventId: "f-1", forgottenAt: "2026-01-01T00:00:00.000Z", reason: "retention", @@ -217,6 +237,7 @@ describe("a forget ledger's source", () => { await ledger?.append([ { personId: "c-1", + categories: null, eventId: "e-1", forgottenAt: "2026-01-01T00:00:00.000Z", reason: "requested", @@ -281,6 +302,7 @@ describe("the wait for the forget ledger", () => { describe("the forget ledger bound by its timeout", () => { const entry = { personId: "c-1", + categories: null, eventId: "e-1", forgottenAt: "2026-01-01T00:00:00.000Z", reason: "requested" as const, diff --git a/packages/core/src/kernel/personal-data/ledger-outbox.ts b/packages/core/src/kernel/personal-data/ledger-outbox.ts index ee1784b3..5c6a97d2 100644 --- a/packages/core/src/kernel/personal-data/ledger-outbox.ts +++ b/packages/core/src/kernel/personal-data/ledger-outbox.ts @@ -13,14 +13,22 @@ export interface IsForgettingFunction { export const isForgetting: IsForgettingFunction = (event) => event.aggregateType === BOUNDA_AGGREGATE && event.type === PERSONAL_DATA_FORGOTTEN_EVENT; +// As `bounda forgotten` and the KV ledger read a list: whole, or not at all. +const isTextList = (value: unknown): value is readonly string[] => + Array.isArray(value) && value.every((item) => typeof item === "string"); + +const names = (value: unknown): readonly string[] => (isTextList(value) ? value : []); + export interface ForgettingEntryFunction { (event: StoredEvent): ForgetLedgerEntry; } /** * The entry of a `PersonalDataForgotten`, as its intention wrote it. An event written before - * forgettings named their keys leaves `fingerprints` out, and one whose reason is no longer known - * reads as `requested`. + * forgettings named their keys leaves `fingerprints` out, one whose reason is no longer known + * reads as `requested`, and one that does not say it forgot only some categories forgot the person + * as a whole. One that did asked for the categories it names, those retained included: one of + * what an earlier forgetting retained asked for none, as the retention ends that again on its own. */ export const forgettingEntry: ForgettingEntryFunction = (event) => { const field = (name: string): unknown => @@ -30,13 +38,15 @@ export const forgettingEntry: ForgettingEntryFunction = (event) => { const fingerprints = field("fingerprints"); const entry: ForgetLedgerEntry = { personId: event.aggregateId, + categories: + field("whole") === false + ? [...names(field("categories")), ...names(field("retained"))] + : null, eventId: event.id, forgottenAt: event.timestamp, reason: field("reason") === "retention" ? "retention" : "requested", }; - return Array.isArray(fingerprints) && fingerprints.every((value) => typeof value === "string") - ? { ...entry, fingerprints } - : entry; + return isTextList(fingerprints) ? { ...entry, fingerprints } : entry; }; export interface LedgerEntriesArgs { diff --git a/packages/core/src/kernel/personal-data/marks.test.ts b/packages/core/src/kernel/personal-data/marks.test.ts index 0b8af6fc..5633b0a0 100644 --- a/packages/core/src/kernel/personal-data/marks.test.ts +++ b/packages/core/src/kernel/personal-data/marks.test.ts @@ -1,10 +1,16 @@ import { describe, expect, it } from "vitest"; import { z } from "zod"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { ConfigurationError } from "../../contracts/errors.ts"; import { person, personal } from "../../modules/personal.ts"; import type { AggregatesRuntime } from "../aggregate/runtime.ts"; import type { ProcessesRuntime } from "../process/build-processes.ts"; -import { marksPersonalFields, type PersonalMarks, personalMarksOf } from "./marks.ts"; +import { + fieldCategory, + marksPersonalFields, + type PersonalMarks, + personalMarksOf, +} from "./marks.ts"; const path = "aggregates.order.events.orderPlaced"; @@ -26,6 +32,12 @@ describe("personalMarksOf", () => { expect(personalMarksOf({ schema, path })).toEqual({ person: null, fields: ["email", "address", "phone", "name"], + categories: { + email: DEFAULT_PERSONAL_CATEGORY, + address: DEFAULT_PERSONAL_CATEGORY, + phone: DEFAULT_PERSONAL_CATEGORY, + name: DEFAULT_PERSONAL_CATEGORY, + }, }); }); @@ -34,7 +46,29 @@ describe("personalMarksOf", () => { customerId: person(z.uuid()).readonly(), note: personal(z.string()), }); - expect(personalMarksOf({ schema, path })).toEqual({ person: "customerId", fields: ["note"] }); + expect(personalMarksOf({ schema, path })).toEqual({ + person: "customerId", + fields: ["note"], + categories: { note: DEFAULT_PERSONAL_CATEGORY }, + }); + }); + + it("tells the category each personal field is sealed under", () => { + const schema = z.object({ + email: personal(z.email(), { category: "profile" }), + iban: personal(z.string(), { category: "invoices" }).optional(), + note: personal(z.string()), + }); + const marks = personalMarksOf({ schema, path }); + expect(marks?.categories).toEqual({ + email: "profile", + iban: "invoices", + note: DEFAULT_PERSONAL_CATEGORY, + }); + if (marks === null) throw new Error("expected marks"); + expect(fieldCategory(marks, "iban")).toBe("invoices"); + expect(fieldCategory(marks, "note")).toBe(DEFAULT_PERSONAL_CATEGORY); + expect(fieldCategory({ ...marks, categories: {} }, "email")).toBe(DEFAULT_PERSONAL_CATEGORY); }); it("finds no mark deeper than the top level of the payload", () => { @@ -44,7 +78,11 @@ describe("personalMarksOf", () => { it("names the person of a payload without personal fields, which nothing encrypts", () => { const schema = z.object({ customerId: person(z.string()), total: z.number() }); - expect(personalMarksOf({ schema, path })).toEqual({ person: "customerId", fields: [] }); + expect(personalMarksOf({ schema, path })).toEqual({ + person: "customerId", + fields: [], + categories: {}, + }); }); it("refuses two person fields", () => { @@ -82,8 +120,12 @@ describe("marksPersonalFields", () => { }) as unknown as AggregatesRuntime; const processes = (states: readonly (PersonalMarks | null)[]): ProcessesRuntime => ({ all: states.map((personal) => ({ personal })) }) as unknown as ProcessesRuntime; - const fields = { person: null, fields: ["email"] }; - const personOnly = { person: "customerId", fields: [] }; + const fields = { + person: null, + fields: ["email"], + categories: { email: DEFAULT_PERSONAL_CATEGORY }, + }; + const personOnly = { person: "customerId", fields: [], categories: {} }; const marks = ( events: readonly (PersonalMarks | null)[], commands: readonly (PersonalMarks | null)[], diff --git a/packages/core/src/kernel/personal-data/marks.ts b/packages/core/src/kernel/personal-data/marks.ts index 5b9dcdd8..451d28ff 100644 --- a/packages/core/src/kernel/personal-data/marks.ts +++ b/packages/core/src/kernel/personal-data/marks.ts @@ -1,4 +1,5 @@ import { z } from "zod"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { ConfigurationError } from "../../contracts/errors.ts"; import { personalMarkOf } from "../../modules/personal.ts"; import type { AggregatesRuntime } from "../aggregate/runtime.ts"; @@ -6,12 +7,14 @@ import type { ProcessesRuntime } from "../process/build-processes.ts"; /** * What an event's or a command's payload schema, or a process's state schema, marks: the fields to - * encrypt, and the field naming the person they belong to, or whom the payload touches when it - * marks no field to encrypt; `null` when the aggregate's or the instance's id names them. + * encrypt, the category each is sealed under, and the field naming the person they belong to, or + * whom the payload touches when it marks no field to encrypt; `null` when the aggregate's or the + * instance's id names them. */ export interface PersonalMarks { readonly person: string | null; readonly fields: readonly string[]; + readonly categories: Readonly>; } export interface PersonalMarksOfArgs { @@ -31,11 +34,15 @@ export interface PersonalMarksOfFunction { export const personalMarksOf: PersonalMarksOfFunction = ({ schema, path }) => { if (!(schema instanceof z.ZodObject)) return null; const fields: string[] = []; + const categories: Record = {}; const persons: string[] = []; for (const [name, field] of Object.entries(schema.shape)) { - const mark = personalMarkOf(field as z.ZodType); - if (mark === "personal") fields.push(name); - if (mark === "person") persons.push(name); + const marked = personalMarkOf(field as z.ZodType); + if (marked?.mark === "personal") { + fields.push(name); + categories[name] = marked.category; + } + if (marked?.mark === "person") persons.push(name); } if (persons.length > 1) { throw new ConfigurationError( @@ -43,9 +50,16 @@ export const personalMarksOf: PersonalMarksOfFunction = ({ schema, path }) => { ); } if (fields.length === 0 && persons.length === 0) return null; - return { person: persons[0] ?? null, fields }; + return { person: persons[0] ?? null, fields, categories }; }; +export interface FieldCategoryFunction { + (marks: PersonalMarks, field: string): string; +} + +export const fieldCategory: FieldCategoryFunction = (marks, field) => + marks.categories[field] ?? DEFAULT_PERSONAL_CATEGORY; + export interface MarksPersonalFieldsArgs { readonly aggregates: AggregatesRuntime; readonly processes: ProcessesRuntime; diff --git a/packages/core/src/kernel/personal-data/personal-data.test.ts b/packages/core/src/kernel/personal-data/personal-data.test.ts index 1d057890..35333925 100644 --- a/packages/core/src/kernel/personal-data/personal-data.test.ts +++ b/packages/core/src/kernel/personal-data/personal-data.test.ts @@ -390,7 +390,7 @@ describe("personal data in an app", () => { await app.commands.registerCustomer(ana); await app.runUntilIdle(); const stream = { aggregateType: "customer", aggregateId: "c-1" }; - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", "default"); expect(await storage.snapshotStore.get(stream)).toMatchObject({ version: 1, persons: [{ person: "c-1", fingerprint: key?.fingerprint }], @@ -434,7 +434,7 @@ describe("personal data in an app", () => { await app.commands.greetCustomer({ customerId: "c-9" }); await app.runUntilIdle(); const stream = { aggregateType: "customer", aggregateId: "c-9" }; - const key = await storage.personalKeys.get("c-9"); + const key = await storage.personalKeys.get("c-9", "default"); const [registered] = await storage.eventStore.readAll({ afterPosition: 0, limit: 1 }); expect(registered).toMatchObject({ person: "c-9", @@ -469,7 +469,7 @@ describe("personal data in an app", () => { rejected: "NotYet", message: "ana@example.com cannot be confirmed yet", }); - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", "default"); expect( await storage.idempotencyStore.get({ key: "confirm-1", now: clock.now() }), ).toMatchObject({ @@ -501,7 +501,7 @@ describe("personal data in an app", () => { expect(reminded.eventTypes).toEqual(["CustomerReminded"]); const [reminder] = await app.events.read({ after: reminded.position - 1 }); expect(reminder?.payload).toEqual({}); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).toBeNull(); expect(JSON.stringify(await app.events.read())).not.toContain("ana@example.com"); await app.stop(); }); @@ -517,7 +517,7 @@ describe("personal data in an app", () => { const { app } = await createTestApp({ registry, ...setup, adapter }); await app.commands.registerCustomer(ana); await app.runUntilIdle(); - const before = await storage.personalKeys.get("c-1"); + const before = await storage.personalKeys.get("c-1", "default"); whileDeciding = async () => { await app.forget("c-1"); await app.commands.subscribe({ @@ -531,7 +531,7 @@ describe("personal data in an app", () => { const [reminder] = await app.events.read({ after: reminded.position - 1 }); expect(reminder?.type).toBe("CustomerReminded"); expect(reminder?.payload).toEqual({}); - const after = await storage.personalKeys.get("c-1"); + const after = await storage.personalKeys.get("c-1", "default"); expect(after).not.toBeNull(); expect(after?.fingerprint).not.toBe(before?.fingerprint); await app.stop(); diff --git a/packages/core/src/kernel/personal-data/persons.test.ts b/packages/core/src/kernel/personal-data/persons.test.ts index 6c46d4c1..e6e799e6 100644 --- a/packages/core/src/kernel/personal-data/persons.test.ts +++ b/packages/core/src/kernel/personal-data/persons.test.ts @@ -112,10 +112,7 @@ describe("personsOf", () => { personsOf({ aggregate, from: [held("c-0", "0"), held("c-1", "old"), held("c-1", "1")], - fingerprints: new Map([ - ["c-1", "f-1"], - ["c-9", "f-9"], - ]), + opened: [held("c-1", "1"), held("c-9", "9")], events: [ event(1, "OrderPlaced", { customerId: "c-1", note: "ring" }), event(2, "OrderSigned", { signature: "x" }), @@ -133,6 +130,18 @@ describe("personsOf", () => { ]); }); + it("names a person once for each key their fields were opened with, a retained one as such", () => { + if (aggregate === undefined) throw new Error("no aggregate"); + const retained = { ...held("c-1", "2"), retained: true as const }; + expect( + personsOf({ + aggregate, + opened: [held("c-1", "1"), retained, held("c-2", "3")], + events: [event(1, "OrderPlaced", { customerId: "c-1", note: "ring", email: "a@b.c" })], + }), + ).toEqual([held("c-1", "1"), retained]); + }); + it("names no one for an event whose personal fields are forgotten or absent", () => { if (aggregate === undefined) throw new Error("no aggregate"); expect( diff --git a/packages/core/src/kernel/personal-data/persons.ts b/packages/core/src/kernel/personal-data/persons.ts index 981034b6..ab19dd85 100644 --- a/packages/core/src/kernel/personal-data/persons.ts +++ b/packages/core/src/kernel/personal-data/persons.ts @@ -6,8 +6,8 @@ import { isRecord } from "../shared/merge-fields.ts"; export interface PersonsOfArgs { readonly aggregate: AggregateRuntime; readonly events: readonly StoredEvent[]; - // The key each person's sealed fields were opened with, as the load that read them reports it. - readonly fingerprints?: ReadonlyMap | undefined; + // The keys each person's sealed fields were opened with, as the load that read them reports them. + readonly opened?: readonly KeyedPerson[] | undefined; // Persons already known, such as a snapshot's, kept first. readonly from?: readonly KeyedPerson[]; } @@ -19,15 +19,11 @@ export interface PersonsOfFunction { /** * The people whose personal data the events carry, once each per key: whoever an event names, by * its person field or its aggregate's id, when one of its marked fields still holds a value, with - * the key it was opened with. A field forgotten reads as `null` and names no one, so a forgotten + * each key their fields were opened with, or with `null` when none was, as for events not sealed + * yet. A field forgotten reads as `null` and names no one, so a forgotten * person's stream can be snapshotted again without them. */ -export const personsOf: PersonsOfFunction = ({ - aggregate, - events, - fingerprints = new Map(), - from = [], -}) => { +export const personsOf: PersonsOfFunction = ({ aggregate, events, opened = [], from = [] }) => { const persons = new Map(); const add = (held: KeyedPerson): void => { persons.set(keyedPersonId(held), held); @@ -43,7 +39,9 @@ export const personsOf: PersonsOfFunction = ({ const named = marks.person === null ? event.aggregateId : payload[marks.person]; if (typeof named !== "string" && typeof named !== "number") continue; const person = String(named); - add({ person, fingerprint: fingerprints.get(person) ?? null }); + const keys = opened.filter((held) => held.person === person); + if (keys.length === 0) add({ person, fingerprint: null }); + for (const held of keys) add(held); } return [...persons.values()]; }; diff --git a/packages/core/src/kernel/personal-data/process-state.test.ts b/packages/core/src/kernel/personal-data/process-state.test.ts index 275bc2cb..36fef6fb 100644 --- a/packages/core/src/kernel/personal-data/process-state.test.ts +++ b/packages/core/src/kernel/personal-data/process-state.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from "vitest"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { ValidationError } from "../../contracts/errors.ts"; import { newStoredKey, openedKey } from "../test-support.ts"; import { isSealed, unseal } from "./crypto.ts"; @@ -11,8 +12,16 @@ import { statePersonOf, } from "./process-state.ts"; -const byField: PersonalMarks = { person: "customerId", fields: ["email", "phone"] }; -const byInstance: PersonalMarks = { person: null, fields: ["email"] }; +const byField: PersonalMarks = { + person: "customerId", + fields: ["email", "phone"], + categories: { email: "profile", phone: DEFAULT_PERSONAL_CATEGORY }, +}; +const byInstance: PersonalMarks = { + person: null, + fields: ["email"], + categories: { email: DEFAULT_PERSONAL_CATEGORY }, +}; describe("lifecycleStateOf", () => { it("is the state a lifecycle event carries, and null for one that carries none", () => { @@ -79,8 +88,8 @@ describe("sealState", () => { marks: byField, state, person: "c-1", - key: async (person) => { - persons.push(person); + key: async (person, category) => { + persons.push(`${person}:${category}`); return key; }, }); @@ -91,7 +100,8 @@ describe("sealState", () => { expect(sealed.nudges).toBe(2); expect(isSealed(sealed.email)).toBe(true); expect(isSealed(sealed.email) && (await unseal({ key, sealed: sealed.email }))).toBe("a@b.c"); - expect(persons).toEqual(["c-1"]); + expect(persons).toEqual(["c-1:profile"]); + expect(isSealed(sealed.email) && sealed.email.$personal.category).toBe("profile"); expect(await sealOnce({ ...sealed, phone: "555" })).toMatchObject({ email: sealed.email }); }); }); diff --git a/packages/core/src/kernel/personal-data/process-state.ts b/packages/core/src/kernel/personal-data/process-state.ts index d90ce93d..cfb5ca83 100644 --- a/packages/core/src/kernel/personal-data/process-state.ts +++ b/packages/core/src/kernel/personal-data/process-state.ts @@ -1,7 +1,7 @@ import { ValidationError } from "../../contracts/errors.ts"; import { isRecord } from "../shared/merge-fields.ts"; import { isSealed, type PersonalKey, seal } from "./crypto.ts"; -import type { PersonalMarks } from "./marks.ts"; +import { fieldCategory, type PersonalMarks } from "./marks.ts"; export interface LifecycleStateOfFunction { (payload: unknown): Readonly> | null; @@ -73,7 +73,7 @@ export interface SealStateArgs { readonly marks: PersonalMarks; readonly state: Readonly>; readonly person: string; - readonly key: (person: string) => Promise; + readonly key: (person: string, category: string) => Promise; } export interface SealStateFunction { @@ -81,7 +81,8 @@ export interface SealStateFunction { } /** - * The state with each personal field that holds a value sealed under the person's key. One that + * The state with each personal field that holds a value sealed under the person's key of the + * field's category. One that * holds `null` holds nothing of theirs and is kept, as is one sealed already, so a lifecycle event * written before its fields were marked can be sealed in place with what it still has in the * clear. @@ -91,7 +92,8 @@ export const sealState: SealStateFunction = async ({ marks, state, person, key } for (const field of marks.fields) { const value = state[field]; if (!holdsValue(value) || isSealed(value)) continue; - sealed[field] = await seal({ key: await key(person), person, value }); + const category = fieldCategory(marks, field); + sealed[field] = await seal({ key: await key(person, category), person, category, value }); } return sealed; }; @@ -100,7 +102,7 @@ export interface SealLifecycleEventArgs { readonly process: string; readonly marks: PersonalMarks; readonly event: Event; - readonly key: (person: string) => Promise; + readonly key: (person: string, category: string) => Promise; } export interface LifecycleEvent { diff --git a/packages/core/src/kernel/personal-data/reads.ts b/packages/core/src/kernel/personal-data/reads.ts index 73740e18..0c4ba937 100644 --- a/packages/core/src/kernel/personal-data/reads.ts +++ b/packages/core/src/kernel/personal-data/reads.ts @@ -1,4 +1,4 @@ -import type { KeyedPerson } from "../../adapter/storage/personal-key-store.ts"; +import { type KeyedPerson, keyedPersonId } from "../../adapter/storage/personal-key-store.ts"; // What the failed run read, by the error it failed with: the error is what reaches whoever files // its dead letter, after the unit of work that read is gone. @@ -28,17 +28,18 @@ export interface ReadUnderArgs { } export interface ReadUnderFunction { - (args: ReadUnderArgs): readonly string[]; + (args: ReadUnderArgs): readonly KeyedPerson[]; } /** - * The fingerprints of the keys the failed work read the person's data under. + * The keys the failed work read the person's data under, each once, as it read them: kept past a + * forgetting, `retained`, or in force. */ export const readUnder: ReadUnderFunction = ({ person, error, also = [] }) => { const noted = readsByError.get(Object(error)) ?? []; - const fingerprints = new Set(); + const reads = new Map(); for (const read of [...noted, ...also]) { - if (read.person === person && read.fingerprint !== null) fingerprints.add(read.fingerprint); + if (read.person === person && read.fingerprint !== null) reads.set(keyedPersonId(read), read); } - return [...fingerprints]; + return [...reads.values()]; }; diff --git a/packages/core/src/kernel/personal-data/recategorizing.test.ts b/packages/core/src/kernel/personal-data/recategorizing.test.ts new file mode 100644 index 00000000..c6110a18 --- /dev/null +++ b/packages/core/src/kernel/personal-data/recategorizing.test.ts @@ -0,0 +1,160 @@ +import { describe, expect, it } from "vitest"; +import type { Adapter } from "../../adapter/adapter.ts"; +import { createNodeSqliteAdapter } from "../../adapter/sqlite/node-sqlite.ts"; +import type { SealedValue } from "../../adapter/storage/personal-key-store.ts"; +import { createFixedClock } from "../../contracts/clock.ts"; +import { createSequentialIdGenerator } from "../../contracts/ids.ts"; +import { silentLogger } from "../../contracts/logger.ts"; +import { memory } from "../../memory/index.ts"; +import type { CommandPayloadArgs, EventPayloadArgs, PayloadArgs } from "../../modules/payload.ts"; +import type { Registry } from "../../modules/registry.ts"; +import { createApp } from "../app.ts"; + +const START = new Date("2026-05-01T00:00:00.000Z"); + +// The email is in `category`; the default one when it is `undefined`. +const registryOf = (category: "marketing" | "billing" | undefined): Registry => + ({ + aggregates: { + customer: { + state: { initialState: {} }, + events: { + customerRegistered: { + payload: ({ z, personal }: EventPayloadArgs) => + z.object({ + email: personal(z.email(), category === undefined ? undefined : { category }), + }), + evolve: () => ({}), + }, + }, + commands: { + registerCustomer: { + module: { + payload: ({ z }: PayloadArgs) => + z.object({ customerId: z.string(), email: z.email() }), + handler: ({ + command, + events, + }: { + command: { payload: { email: string } }; + events: Record unknown>; + }) => [events.customerRegistered?.({ email: command.payload.email })], + }, + }, + remindCustomer: { + module: { + payload: ({ z, personal }: CommandPayloadArgs) => + z.object({ + customerId: z.string(), + email: personal(z.email(), category === undefined ? undefined : { category }), + }), + handler: () => [], + }, + }, + }, + policies: {}, + processes: {}, + }, + }, + readModels: {}, + personalData: { + categories: { + marketing: { basis: "consent", retention: "2y" }, + billing: { basis: "contract", retention: "2y" }, + }, + }, + }) as unknown as Registry; + +let starts = 0; + +// Each start a minute after the one before, as deploys follow one another. +const start = async (adapter: Adapter, category: "marketing" | "billing" | undefined) => { + starts += 1; + const clock = createFixedClock(new Date(START.getTime() + starts * 60_000)); + const app = await createApp({ + registry: registryOf(category), + config: { storage: adapter }, + ids: createSequentialIdGenerator({ prefix: `app-${starts}` }), + clock, + logger: silentLogger, + }); + return app; +}; + +const sealedEmail = async (adapter: Adapter) => { + const storage = await adapter.createStorage(); + const [event] = await storage.eventStore.readAll({ afterPosition: 0, limit: 10 }); + const [scheduled] = await storage.scheduler.list(); + return { + event: (event?.payload as { email: SealedValue } | undefined)?.email.$personal, + scheduled: (scheduled?.command.payload as { email?: SealedValue } | undefined)?.email + ?.$personal, + persons: scheduled?.persons, + }; +}; + +const adapters: readonly [string, () => Adapter][] = [ + ["memory", () => memory()], + ["SQLite (node:sqlite)", () => createNodeSqliteAdapter().adapter], +]; + +describe.each(adapters)("a field moved to another category, on %s", (_name, adapterOf) => { + it("is sealed anew under its new category, so forgetting the old one no longer reaches it and forgetting the new one does", async () => { + const adapter = adapterOf(); + const before = await start(adapter, "marketing"); + await before.commands.registerCustomer?.({ customerId: "c-1", email: "ada@example.com" }); + await before.commands.remindCustomer?.( + { customerId: "c-1", email: "ada@example.com" }, + { delay: "1d" }, + ); + await before.runUntilIdle(); + await before.stop(); + expect((await sealedEmail(adapter)).event?.category).toBe("marketing"); + + const after = await start(adapter, "billing"); + await after.runUntilIdle(); + const moved = await sealedEmail(adapter); + expect(moved.event?.category).toBe("billing"); + expect(moved.scheduled?.category).toBe("billing"); + expect(moved.persons).toEqual([{ person: "c-1", fingerprint: moved.scheduled?.key }]); + await after.forget("c-1", { categories: ["marketing"] }); + const [read] = await after.events.read({ limit: 1 }); + expect((await after.exportPersonalData("c-1")).events[0]?.data).toEqual({ + email: "ada@example.com", + }); + expect(read?.type).toBe("CustomerRegistered"); + expect((await adapter.createStorage().then((s) => s.scheduler.list())).length).toBe(1); + await after.forget("c-1", { categories: ["billing"] }); + expect((await after.exportPersonalData("c-1")).events[0]?.data).toEqual({ email: null }); + expect((await adapter.createStorage().then((s) => s.scheduler.list())).length).toBe(0); + await after.stop(); + }); + + it("moves to and from the default category, and leaves what no key opens any more as it is", async () => { + const adapter = adapterOf(); + const before = await start(adapter, undefined); + await before.commands.registerCustomer?.({ customerId: "c-1", email: "ada@example.com" }); + await before.commands.registerCustomer?.({ customerId: "c-2", email: "bo@example.com" }); + await before.forget("c-2"); + await before.runUntilIdle(); + await before.stop(); + + const marketing = await start(adapter, "marketing"); + await marketing.runUntilIdle(); + const storage = await adapter.createStorage(); + const events = await storage.eventStore.readAll({ afterPosition: 0, limit: 10 }); + const emails = events + .filter(({ type }) => type === "CustomerRegistered") + .map(({ payload }) => (payload as { email: SealedValue }).email.$personal.category); + expect(emails).toEqual(["marketing", undefined]); + await marketing.stop(); + + const back = await start(adapter, undefined); + await back.runUntilIdle(); + expect((await sealedEmail(adapter)).event?.category).toBeUndefined(); + expect((await back.exportPersonalData("c-1")).events[0]?.data).toEqual({ + email: "ada@example.com", + }); + await back.stop(); + }); +}); diff --git a/packages/core/src/kernel/personal-data/reconciliation-app.test.ts b/packages/core/src/kernel/personal-data/reconciliation-app.test.ts index a61dd29c..6a8b9e42 100644 --- a/packages/core/src/kernel/personal-data/reconciliation-app.test.ts +++ b/packages/core/src/kernel/personal-data/reconciliation-app.test.ts @@ -183,8 +183,14 @@ const start = async ({ return { app, clock, register, forgotten, customer }; }; -const entry = (personId: string, eventId: string, at: Date): ForgetLedgerEntry => ({ +const entry = ( + personId: string, + eventId: string, + at: Date, + categories: readonly string[] | null = null, +): ForgetLedgerEntry => ({ personId, + categories, eventId, forgottenAt: at.toISOString(), reason: "requested", @@ -242,7 +248,7 @@ describe("the forget ledger", () => { expect(logs).toContainEqual([ "info", "personal data forgotten", - expect.objectContaining({ personId: "c-1", reason: "requested", hadKey: true }), + expect.objectContaining({ personId: "c-1", reason: "requested", destroyedKeys: 1 }), ]); const exported = await app.exportPersonalData("c-1"); expect(exported.events.map(({ type, data }) => [type, data])).toEqual([ @@ -668,6 +674,37 @@ describe("the forget ledger", () => { expect(await (await opened(ledgerAdapter)).read({ limit: 10 })).toEqual(appended); }); + it("keeps the categories a forgetting asked for, and forgets those again on a restore", async () => { + const storage = memory(); + const ledgerAdapter = memory(); + const { app: before, register } = await start({ storage, forgetLedger: ledgerAdapter }); + await register("c-1"); + await before.forget("c-1", { categories: ["default"] }); + const [forgetting] = (await before.events.read({ limit: 100 })).filter( + (event) => event.type === "PersonalDataForgotten", + ); + expect(await (await opened(ledgerAdapter)).read({ limit: 10 })).toEqual([ + { + ...entry("c-1", forgetting?.id ?? "", START, ["default"]), + fingerprints: [expect.any(String)], + }, + ]); + await register("c-2"); + await (await opened(ledgerAdapter)).append([entry("c-2", "lost-forget", LOST, ["default"])]); + await before.stop(); + const { app, forgotten, customer } = await start({ storage, forgetLedger: ledgerAdapter }); + expect(await forgotten()).toEqual([ + ["c-1", "requested"], + ["c-2", "requested"], + ]); + expect(await customer("c-2")).toBeNull(); + const restored = (await app.events.read({ limit: 100 })).at(-1); + expect(restored).toMatchObject({ + id: "lost-forget", + payload: { whole: false, categories: ["default"] }, + }); + }); + it("appends the forgettings of forgetAgain as they commit", async () => { const storage = memory(); const ledgerAdapter = memory(); @@ -1079,6 +1116,7 @@ describe("the intention a forgetting records first", () => { const [intention] = await (await opened(ledgerAdapter)).read({ limit: 10 }); expect(intention).toEqual({ personId: "c-1", + categories: null, eventId: expect.any(String), forgottenAt: START.toISOString(), reason: "requested", @@ -1275,7 +1313,7 @@ describe("the intention a forgetting records first", () => { expect(await forgettings()).toEqual([]); expect(logs).toContainEqual([ "error", - "person past the retention could not be forgotten", + "personal data past the retention could not be forgotten", expect.objectContaining({ personId: "c-1", errorCode: "FORGET_LEDGER_UNAVAILABLE" }), ]); down = false; diff --git a/packages/core/src/kernel/personal-data/reconciliation.test.ts b/packages/core/src/kernel/personal-data/reconciliation.test.ts index 9b46f630..944aff50 100644 --- a/packages/core/src/kernel/personal-data/reconciliation.test.ts +++ b/packages/core/src/kernel/personal-data/reconciliation.test.ts @@ -6,12 +6,14 @@ import type { ForgetLedgerEntry, ReadForgetLedgerArgs, } from "../../adapter/storage/forget-ledger.ts"; +import { testPersonalKey } from "../../adapter/testing/fixtures.ts"; import { createFixedClock } from "../../contracts/clock.ts"; import { ForgetLedgerUnavailableError, ValidationError } from "../../contracts/errors.ts"; import { createSequentialIdGenerator } from "../../contracts/ids.ts"; import { type LogFields, type Logger, silentLogger } from "../../contracts/logger.ts"; import { memory } from "../../memory/index.ts"; import { withoutLedger } from "../test-support.ts"; +import { buildPersonalCategories } from "./categories.ts"; import { boundForgetLedger } from "./forget-ledger.ts"; import { HOLD_MS } from "./hold.ts"; import { createForgetIntentions } from "./intention.ts"; @@ -37,12 +39,13 @@ const loggingErrors = (errors: unknown[]): Logger => ({ }); const NOW = new Date("2026-03-01T00:00:00.000Z"); +const NO_CATEGORIES = buildPersonalCategories({ module: undefined, retention: null }); const EVERY = 5 * 60_000; const keyed = async (storage: Storage, persons: readonly string[]): Promise => { for (const person of persons) { await storage.transact((transaction) => - transaction.personalKeys.add(person, { + transaction.personalKeys.add(person, "default", { key: `key-${person}`, fingerprint: `f-${person}`, kid: null, @@ -94,6 +97,8 @@ const setUp = async ({ clock: createFixedClock(NOW), logger, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs: null, pageSize, @@ -105,6 +110,7 @@ const entry = (personId: string, eventId: string, forgottenAt: string): ForgetLe personId, eventId, forgottenAt, + categories: null, reason: "requested", }); @@ -232,6 +238,8 @@ describe("the forget ledger's reconciliation", () => { clock: createFixedClock(NOW), logger: silentLogger, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs: null, pageSize: 10, @@ -280,6 +288,8 @@ describe("the forget ledger's reconciliation", () => { clock: createFixedClock(NOW), logger: loggingErrors(errors), concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs: null, }); @@ -327,6 +337,8 @@ describe("the forget ledger's reconciliation", () => { clock: createFixedClock(NOW), logger: loggingErrors(errors), concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs: null, }); @@ -367,6 +379,8 @@ describe("the forget ledger's reconciliation", () => { clock: createFixedClock(NOW), logger: silentLogger, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs: null, }); @@ -396,6 +410,8 @@ describe("the forget ledger's reconciliation", () => { clock: createFixedClock(NOW), logger: silentLogger, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs: null, forgotten: (personId) => told.push(personId), @@ -458,6 +474,8 @@ describe("the forget ledger's periodic reconciliation", () => { clock: createFixedClock(NOW), logger, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs, }); @@ -633,7 +651,7 @@ describe("what the reconciliation commits again", () => { persons: ["c-1"], }); expect(await reconciliation.reconcile(NOW)).toBe(forgotten); - expect((await storage.personalKeys.get("c-1")) === null).toBe(forgotten === 1); + expect((await storage.personalKeys.get("c-1", "default")) === null).toBe(forgotten === 1); } }); @@ -674,6 +692,8 @@ describe("what the reconciliation commits again", () => { clock, logger: silentLogger, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs: null, }); @@ -699,7 +719,7 @@ describe("what the reconciliation commits again", () => { persons: ["c-1", "c-2", "c-3"], }); expect(await reconciliation.reconcile(NOW)).toBe(1); - expect(await storage.personalKeys.get("c-2")).not.toBeNull(); + expect(await storage.personalKeys.get("c-2", "default")).not.toBeNull(); expect(reconciliation.nextDueAt()).toEqual( new Date(Date.parse(younger) + FORGET_LEDGER_GRACE_MS), ); @@ -717,7 +737,7 @@ describe("what the reconciliation commits again", () => { expect(await reconciliation.reconcile(NOW)).toBe(1); entries.push(entry("c-2", "e-2", young)); expect(await reconciliation.reconcile(NOW)).toBe(0); - expect(await storage.personalKeys.get("c-2")).not.toBeNull(); + expect(await storage.personalKeys.get("c-2", "default")).not.toBeNull(); }); it("is due at the sooner of a ledger's delay and an entry's grace", async () => { @@ -747,6 +767,8 @@ describe("forgetAgain", () => { ids, clock, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, entry: { personId: "" }, retentionDue: NEVER_DUE, record: true, @@ -774,6 +796,8 @@ describe("forgetAgain", () => { ids, clock, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, entry, retentionDue: NEVER_DUE, record: true, @@ -785,6 +809,125 @@ describe("forgetAgain", () => { expect(transactions).toBe(1); }); + it("forgets again only the categories the entry asked for, while one of them still has a key in force", async () => { + const storage = await memory().createStorage(); + const categories = buildPersonalCategories({ + module: { + categories: { + profile: { basis: "consent", retention: "30d" }, + invoices: { basis: "legal-obligation", retention: "30d" }, + }, + }, + retention: null, + }); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "profile", testPersonalKey("2")); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("3")); + await storage.personalKeys.retain("c-1", "f-3", { since: at(0), lastActiveAt: at(0) }); + const ids = createSequentialIdGenerator(); + const clock = createFixedClock(NOW); + const run = (asked: readonly string[] | null) => + forgetAgain({ + storage, + intentions: withoutLedger({ ids, clock }), + ids, + clock, + concurrencyRetries: 3, + categories, + batchSize: 10, + entry: { personId: "c-1", eventId: "e-1", categories: asked }, + retentionDue: NEVER_DUE, + record: false, + }); + expect(await run([])).toBeNull(); + expect(await run(["orders"])).toBeNull(); + expect(await run(["invoices"])).toBeNull(); + expect((await run(["profile", "orders"]))?.payload).toMatchObject({ + reason: "requested", + whole: false, + categories: ["profile"], + retained: [], + }); + expect((await storage.personalKeys.list("c-1")).map(({ category }) => category)).toEqual([ + "default", + "invoices", + ]); + expect(await run(["profile"])).toBeNull(); + expect((await run(null))?.payload).toMatchObject({ + whole: true, + categories: ["default", "profile"], + }); + expect((await storage.personalKeys.list("c-1")).map(({ category }) => category)).toEqual([ + "invoices", + ]); + expect(await run(null)).toBeNull(); + }); + + it("commits again a category's retention only while it would still end the keys it names, destroying those alone", async () => { + const storage = await memory().createStorage(); + const categories = buildPersonalCategories({ + module: { + categories: { + profile: { basis: "consent", retention: "30d" }, + invoices: { basis: "legal-obligation", retention: "30d" }, + }, + }, + retention: null, + }); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("3")); + const asked: (readonly string[] | null)[] = []; + let due = false; + const ids = createSequentialIdGenerator(); + const clock = createFixedClock(NOW); + const run = (entry: ForgetAgainEntry) => + forgetAgain({ + storage, + intentions: withoutLedger({ ids, clock }), + ids, + clock, + concurrencyRetries: 3, + categories, + batchSize: 10, + entry, + retentionDue: async (_stores, _person, _now, keys) => { + asked.push(keys === null ? null : keys.map(({ key }) => key.fingerprint)); + return due; + }, + record: false, + }); + const entry: ForgetAgainEntry = { + personId: "c-1", + eventId: "e-1", + forgottenAt: at(0), + reason: "retention", + categories: ["invoices"], + fingerprints: ["f-3"], + }; + expect(await run(entry)).toBeNull(); + expect(asked).toEqual([["f-3"]]); + due = true; + expect( + (await run({ personId: "c-1", reason: "retention", categories: ["invoices"] }))?.payload, + ).toMatchObject({ + reason: "retention", + whole: false, + categories: ["invoices"], + retained: [], + fingerprints: ["f-3"], + }); + expect((await storage.personalKeys.list("c-1")).map(({ category }) => category)).toEqual([ + "default", + ]); + expect(await run(entry)).toBeNull(); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("4")); + asked.length = 0; + expect((await run({ personId: "c-1", reason: "retention" }))?.payload).toMatchObject({ + whole: true, + }); + expect(asked).toEqual([null, null]); + }); + const ledgered = async () => { const base = await memory().createStorage(); let transactions = 0; @@ -829,6 +972,8 @@ describe("forgetAgain", () => { ids, clock, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, entry, retentionDue: NEVER_DUE, record: true, @@ -869,6 +1014,7 @@ describe("forgetAgain", () => { expect(recorded).toEqual([ { personId: "c-1", + categories: null, eventId: "id-2", forgottenAt: NOW.toISOString(), reason: "requested", @@ -885,7 +1031,7 @@ describe("forgetAgain", () => { ForgetLedgerUnavailableError, ); await expect(run({ personId: "c-1" })).rejects.toBeInstanceOf(ForgetLedgerUnavailableError); - expect(await storage.personalKeys.get("c-1")).not.toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).not.toBeNull(); expect(await storage.eventStore.lastPosition()).toBe(0); }); @@ -896,7 +1042,7 @@ describe("forgetAgain", () => { expect(await run({ ...entry("c-1", "e-1", at(3)), fingerprints: ["f-c-1"] })).not.toBeNull(); await keyed(storage, ["c-1"]); expect(await run({ ...entry("c-1", "e-1", at(3)), fingerprints: ["f-c-1"] })).toBeNull(); - expect(await storage.personalKeys.get("c-1")).not.toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).not.toBeNull(); }); it("leaves a key the entry does not name without a transaction", async () => { @@ -915,7 +1061,7 @@ describe("forgetAgain", () => { expect(await run(legacy)).not.toBeNull(); await keyed(storage, ["c-1"]); expect(await run(legacy)).toBeNull(); - expect(await storage.personalKeys.get("c-1")).not.toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).not.toBeNull(); }); it("takes an intention of its own for an entry missing its id, its time or its reason", async () => { @@ -943,8 +1089,12 @@ describe("forgetAgain", () => { const { storage, recorded, run, beforeTransaction } = await ledgered(); await keyed(storage, ["c-1"]); beforeTransaction(async () => { - await storage.personalKeys.remove("c-1"); - await storage.personalKeys.add("c-1", { key: "k-2", fingerprint: "f-2", kid: null }); + await storage.personalKeys.remove("c-1", "f-c-1"); + await storage.personalKeys.add("c-1", "default", { + key: "k-2", + fingerprint: "f-2", + kid: null, + }); }); expect(await run({ personId: "c-1" })).toMatchObject({ event: { id: "id-3", payload: { fingerprints: ["f-2"] } }, @@ -967,6 +1117,8 @@ describe("forgetAgain", () => { clock, logger: silentLogger, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs: null, }); @@ -975,14 +1127,14 @@ describe("forgetAgain", () => { }); it("never commits again a forgetting the retention made without a retention", async () => { - expect(await NEVER_DUE(await memory().createStorage(), "c-1", NOW)).toBe(false); + expect(await NEVER_DUE(await memory().createStorage(), "c-1", NOW, null)).toBe(false); }); it("forgets nobody whose key went between its check and its transaction", async () => { const { storage, run, beforeTransaction } = await ledgered(); await keyed(storage, ["c-1"]); beforeTransaction(async () => { - await storage.personalKeys.remove("c-1"); + await storage.personalKeys.remove("c-1", "f-c-1"); }); expect(await run({ ...entry("c-1", "e-1", at(3)), fingerprints: ["f-c-1"] })).toBeNull(); expect(await storage.eventStore.lastPosition()).toBe(0); @@ -1062,6 +1214,8 @@ describe("the reconciliation's status and restores", () => { clock: createFixedClock(NOW), logger: { ...silentLogger, error: (...args: unknown[]) => logged.push(args) }, concurrencyRetries: 3, + categories: NO_CATEGORIES, + batchSize: 10, reconcileEveryMs: EVERY, keepForMs: null, }); diff --git a/packages/core/src/kernel/personal-data/reconciliation.ts b/packages/core/src/kernel/personal-data/reconciliation.ts index 6440189b..b46c9d1b 100644 --- a/packages/core/src/kernel/personal-data/reconciliation.ts +++ b/packages/core/src/kernel/personal-data/reconciliation.ts @@ -5,14 +5,22 @@ import type { ForgetLedgerPosition, } from "../../adapter/storage/forget-ledger.ts"; import { type Hold, isReleasedBy } from "../../adapter/storage/hold.ts"; +import type { HeldPersonalKey } from "../../adapter/storage/personal-key-store.ts"; import type { Clock } from "../../contracts/clock.ts"; import type { IdGenerator } from "../../contracts/ids.ts"; import type { Logger } from "../../contracts/logger.ts"; import { BOUNDA_AGGREGATE, type PersonalDataForgottenReason } from "../../modules/bounda-events.ts"; import { startedBy } from "../shared/causation.ts"; -import { type Forgotten, forgetWithin, logForgotten, STALE_INTENTION } from "./forget.ts"; +import type { PersonalCategories } from "./categories.ts"; +import { + type ForgetScope, + type Forgotten, + forgetWithin, + logForgotten, + STALE_INTENTION, +} from "./forget.ts"; import { holdAfter } from "./hold.ts"; -import type { ForgetIntention, ForgetIntentions } from "./intention.ts"; +import { coveredBy, type ForgetIntention, type ForgetIntentions } from "./intention.ts"; import { checkPersonId, transactRetrying } from "./transact.ts"; /** @@ -72,17 +80,25 @@ export interface ForgetAgainEntry { * after, and is kept. Any key counts when absent. */ readonly fingerprints?: readonly string[]; + /** + * The categories the forgetting asked for: `null`, or left out, for the person as a whole; none + * for a forgetting of what an earlier one retained, which the retention ends again on its own. + * Only those are forgotten again, never the person as a whole. + */ + readonly categories?: readonly string[] | null; } /** - * Whether the retention would forget the person now, read on the stores given: a forgetting the - * retention made is committed again only then. + * Whether the retention would forget now, read on the stores given, the person as a whole for + * `keys` `null`, else every one of those keys: a forgetting the retention made is committed again + * only then. */ export interface RetentionDueFunction { ( stores: Pick, personId: string, now: Date, + keys: readonly HeldPersonalKey[] | null, ): Promise; } @@ -94,6 +110,8 @@ export interface ForgetAgainArgs { readonly ids: IdGenerator; readonly clock: Clock; readonly concurrencyRetries: number; + readonly categories: PersonalCategories; + readonly batchSize: number; readonly entry: ForgetAgainEntry; readonly retentionDue: RetentionDueFunction; /** @@ -111,13 +129,14 @@ export interface ForgetAgainFunction { type DueStores = Pick; /** - * Commits the forgetting again when the store does not hold the event the entry names, the person - * still holds a key it names, and, for one the retention made, the retention would forget them - * now; deciding all of it again in the transaction that forgets them, so of two instances doing - * it at once, the one whose append conflicts finds it done on its retry. An entry that reads as - * applied takes no transaction. A whole entry is committed under its own id, time and reason; any - * other takes an intention of its own, as a request. Resolves `null` when it forgot nobody. - * `storage` is the store as written: nothing here reads a personal field. + * Commits the forgetting again, of the categories the entry asked for, when the store does not + * hold the event the entry names, the person still holds a key in force it covers and names, and, + * for one the retention made, the retention would forget them now; deciding all of it again in the + * transaction that forgets them, so of two instances doing it at once, the one whose append + * conflicts finds it done on its retry. An entry that reads as applied takes no transaction. A + * complete entry is committed under its own id, time and reason; any other takes an intention of + * its own, as a request. A category the app no longer has has no key to forget. Resolves `null` + * when it forgot nobody. `storage` is the store as written: nothing here reads a personal field. */ export const forgetAgain: ForgetAgainFunction = async ({ storage, @@ -125,58 +144,81 @@ export const forgetAgain: ForgetAgainFunction = async ({ ids, clock, concurrencyRetries, + categories, + batchSize, entry, retentionDue, record, actor, }) => { - const { personId, eventId, forgottenAt, reason, fingerprints } = entry; + const { personId, eventId, forgottenAt, reason, fingerprints, categories: asked = null } = entry; checkPersonId(personId, "forgetting again"); - const isDue = async (stores: DueStores, named: readonly string[] | undefined) => { + const known = asked?.filter((category) => categories.of(category) !== undefined) ?? null; + if (known !== null && known.length === 0) return null; + const asScope: ForgetScope = + known === null ? { kind: "whole" } : { kind: "categories", categories: known }; + // The keys in force the entry covers and names, `null` when it is not due. + const dueKeys = async (stores: DueStores): Promise => { if (eventId !== undefined) { const applied = await stores.eventStore.readOne({ aggregateType: BOUNDA_AGGREGATE, aggregateId: personId, eventId, }); - if (applied !== null) return false; + if (applied !== null) return null; } - const key = await stores.personalKeys.get(personId); - if (key === null || (named !== undefined && !named.includes(key.fingerprint))) return false; - return reason !== "retention" || retentionDue(stores, personId, clock.now()); + const covered = coveredBy(asScope, await stores.personalKeys.list(personId)).filter( + ({ key }) => fingerprints === undefined || fingerprints.includes(key.fingerprint), + ); + if (covered.length === 0) return null; + if (reason !== "retention") return covered; + return (await retentionDue(stores, personId, clock.now(), known === null ? null : covered)) + ? covered + : null; }; - if (!(await isDue(storage, fingerprints))) return null; - const whole = eventId !== undefined && forgottenAt !== undefined && reason !== undefined; - if (whole && record) { - const recorded = { personId, eventId, forgottenAt, reason }; + // A category's retention destroys the keys it found due, whatever their category's basis. + const scopeOf = (covered: readonly HeldPersonalKey[]): ForgetScope => + reason === "retention" && known !== null + ? { kind: "keys", fingerprints: covered.map(({ key }) => key.fingerprint) } + : asScope; + const due = await dueKeys(storage); + if (due === null) return null; + const complete = eventId !== undefined && forgottenAt !== undefined && reason !== undefined; + if (complete && record) { + const recorded = { personId, categories: asked, eventId, forgottenAt, reason }; await intentions.record(fingerprints === undefined ? recorded : { ...recorded, fingerprints }); } const correlationId = ids.next(); - // A whole entry is due only while the person holds a key it names, so it is never stale; an + // A complete entry is due only while the person holds a key it names, so it is never stale; an // intention of its own takes another that names a key given meanwhile, as `forget` does. for (;;) { - const intention: ForgetIntention = whole + const intention: ForgetIntention = complete ? { eventId, forgottenAt, reason, fingerprints } : await intentions.intend({ personId, reason: reason ?? "requested", + scope: scopeOf(due), keys: storage.personalKeys, }); const outcome = await transactRetrying({ storage, concurrencyRetries, - work: async (transaction) => - (await isDue(transaction, fingerprints)) - ? forgetWithin({ - transaction, - ids, - clock, - personId, - intention, - context: startedBy(correlationId, actor), - checked: whole, - }) - : null, + work: async (transaction) => { + const covered = await dueKeys(transaction); + if (covered === null) return null; + return forgetWithin({ + transaction, + ids, + clock, + personId, + intention, + context: startedBy(correlationId, actor), + categories, + scope: scopeOf(covered), + batchSize, + checked: complete, + }); + }, }); if (outcome !== STALE_INTENTION) return outcome === null || "held" in outcome ? null : outcome; } @@ -245,6 +287,8 @@ export interface CreateForgetLedgerReconciliationArgs { readonly ledger: ForgetLedger; readonly intentions: ForgetIntentions; readonly retentionDue: RetentionDueFunction; + readonly categories: PersonalCategories; + readonly batchSize: number; /** * The store as written. */ @@ -291,6 +335,8 @@ export const createForgetLedgerReconciliation: CreateForgetLedgerReconciliationF clock, logger, concurrencyRetries, + categories, + batchSize, reconcileEveryMs, keepForMs, pageSize = FORGET_LEDGER_PAGE, @@ -347,6 +393,8 @@ export const createForgetLedgerReconciliation: CreateForgetLedgerReconciliationF ids, clock, concurrencyRetries, + categories, + batchSize, entry, retentionDue, record: false, diff --git a/packages/core/src/kernel/personal-data/reseal-batches.test.ts b/packages/core/src/kernel/personal-data/reseal-batches.test.ts index 97d89728..a89d4d2e 100644 --- a/packages/core/src/kernel/personal-data/reseal-batches.test.ts +++ b/packages/core/src/kernel/personal-data/reseal-batches.test.ts @@ -2,13 +2,14 @@ import { describe, expect, it } from "vitest"; import type { Storage } from "../../adapter/adapter.ts"; import type { NewDeadLetter } from "../../adapter/storage/dead-letter-store.ts"; import type { PendingEvent } from "../../adapter/storage/event-store.ts"; +import type { SealedValue } from "../../adapter/storage/personal-key-store.ts"; import { createFixedClock } from "../../contracts/clock.ts"; import { memory } from "../../memory/index.ts"; import { BOUNDA_AGGREGATE, PERSONAL_DATA_FORGOTTEN_EVENT } from "../../modules/bounda-events.ts"; import type { AggregatesRuntime } from "../aggregate/runtime.ts"; import type { ProcessesRuntime } from "../process/build-processes.ts"; import { newStoredKey, openedKey } from "../test-support.ts"; -import { isSealed, unseal } from "./crypto.ts"; +import { isSealed, seal, unseal } from "./crypto.ts"; import { createResealBatches } from "./reseal-batches.ts"; import { createPersonalKeyWrapping } from "./wrapped-keys.ts"; @@ -29,24 +30,40 @@ const aggregates = { customer: { ...customer, eventsByType: { - CustomerRegistered: { personal: { person: null, fields: ["email"] } }, + CustomerRegistered: { personal: { person: null, fields: ["email"], categories: {} } }, + CustomerProfiled: { + personal: { + person: null, + fields: ["email", "nickname"], + categories: { nickname: "profile" }, + }, + }, }, }, order: { ...order, eventsByType: { - OrderPlaced: { personal: { person: "customerId", fields: ["address"] } }, + OrderPlaced: { personal: { person: "customerId", fields: ["address"], categories: {} } }, }, }, }, commandsByType: { - RemindCustomer: { command: { personal: { person: null, fields: ["email"] } } }, - ShipOrder: { command: { personal: { person: "customerId", fields: ["address"] } } }, + RemindCustomer: { command: { personal: { person: null, fields: ["email"], categories: {} } } }, + NameCustomer: { + command: { + personal: { person: null, fields: ["nickname"], categories: { nickname: "profile" } }, + }, + }, + ShipOrder: { + command: { personal: { person: "customerId", fields: ["address"], categories: {} } }, + }, }, } as unknown as AggregatesRuntime; const processes = { - byName: { "signup.onboarding": { personal: { person: "customerId", fields: ["email"] } } }, + byName: { + "signup.onboarding": { personal: { person: "customerId", fields: ["email"], categories: {} } }, + }, } as unknown as Pick; const event = ( @@ -67,8 +84,11 @@ const event = ( person: null, }); -const forgotten = (person: string, version = 1): PendingEvent => ({ - ...event(BOUNDA_AGGREGATE, person, version, PERSONAL_DATA_FORGOTTEN_EVENT, { personId: person }), +const forgotten = (person: string, version = 1, payload: object = {}): PendingEvent => ({ + ...event(BOUNDA_AGGREGATE, person, version, PERSONAL_DATA_FORGOTTEN_EVENT, { + personId: person, + ...payload, + }), timestamp: "2026-01-05T00:00:00.000Z", metadata: { ...metadata, system: true }, person, @@ -98,10 +118,16 @@ const setup = async (recordsActivity = true) => { }); } }; - const opened = async (person: string, value: unknown) => { - const stored = await storage.personalKeys.get(person); - const key = stored === null ? null : await openedKey(stored); - return unseal({ key, sealed: value as never }); + // Opens a value sealed under one key or several, one inside the other. + const opened = async (value: unknown): Promise => { + let current = value; + while (isSealed(current)) { + const { person, key: fingerprint } = (current as SealedValue).$personal; + const found = await storage.personalKeys.find(person, fingerprint); + const key = found === null ? null : await openedKey(found.key); + current = await unseal({ key, sealed: current as SealedValue }); + } + return current; }; return { storage, reseal, append, opened }; }; @@ -145,10 +171,10 @@ describe("resealing a batch of events", () => { "c-1", null, ]); - expect(await opened("c-1", fieldOf(ada?.payload, "email"))).toBe("ada@example.com"); - expect(await opened("c-1", fieldOf(placed?.payload, "address"))).toBe("Bilbao"); + expect(await opened(fieldOf(ada?.payload, "email"))).toBe("ada@example.com"); + expect(await opened(fieldOf(placed?.payload, "address"))).toBe("Bilbao"); expect(isSealed(fieldOf(bob?.payload, "email"))).toBe(true); - expect(await storage.personalKeys.get("c-2")).toBeNull(); + expect(await storage.personalKeys.get("c-2", "default")).toBeNull(); expect(nameless?.payload).toEqual({ customerId: "", address: "Getxo" }); expect( await storage.snapshotStore.get({ aggregateType: "customer", aggregateId: "c-1" }), @@ -191,6 +217,119 @@ describe("resealing a batch of events", () => { }); }); +describe("resealing a batch of events by category", () => { + it("seals each field under its category's key, discarding only the key of a category forgotten after it", async () => { + const { storage, reseal, append, opened } = await setup(); + await append( + event("customer", "c-1", 1, "CustomerProfiled", { + email: "ada@example.com", + nickname: "Ada", + }), + forgotten("c-1", 1, { whole: false, categories: ["profile"], retained: [], expired: [] }), + ); + expect(await reseal.events(await all(storage))).toEqual({ + resealed: 1, + discarded: 1, + unsealable: 0, + discardedTypes: new Set(["customer.CustomerProfiled"]), + unsealableTypes: new Set(), + }); + const [profiled] = await all(storage); + expect(await opened(fieldOf(profiled?.payload, "email"))).toBe("ada@example.com"); + expect(fieldOf(profiled?.payload, "nickname")).toMatchObject({ + $personal: { category: "profile" }, + }); + expect(await opened(fieldOf(profiled?.payload, "nickname"))).toBeNull(); + expect((await storage.personalKeys.list("c-1")).map(({ category }) => category)).toEqual([ + "default", + ]); + }); +}); + +describe("resealing a field moved to another category", () => { + it("seals it anew under its category's key, and drops what remembers it under the old key", async () => { + const { storage, reseal, append, opened } = await setup(); + const stored = await storage.personalKeys.add("c-1", "default", await newStoredKey()); + const old = await openedKey(stored); + const nickname = await seal({ key: old, person: "c-1", category: "default", value: "Ada" }); + await append( + event("customer", "c-1", 1, "CustomerProfiled", { email: "ada@example.com", nickname }), + ); + const now = new Date("2026-01-01T00:00:00.000Z"); + await storage.idempotencyStore.put({ + key: "k-1", + fingerprint: "fp", + outcome: { rejected: "Taken", message: "Ada is taken", code: "TAKEN" } as never, + expiresAt: "2099-01-01T00:00:00.000Z", + persons: [{ person: "c-1", fingerprint: stored.fingerprint }], + now, + }); + const extra = await storage.personalKeys.add("c-1", "extra", await newStoredKey()); + await storage.idempotencyStore.put({ + key: "k-2", + fingerprint: "fp", + outcome: { rejected: "Taken", message: "taken", code: "TAKEN" } as never, + expiresAt: "2099-01-01T00:00:00.000Z", + persons: [{ person: "c-1", fingerprint: extra.fingerprint }], + now, + }); + expect((await reseal.events(await all(storage))).resealed).toBe(1); + const [profiled] = await all(storage); + expect(fieldOf(profiled?.payload, "nickname")).toMatchObject({ + $personal: { category: "profile" }, + }); + expect(await opened(fieldOf(profiled?.payload, "nickname"))).toBe("Ada"); + expect(await storage.idempotencyStore.get({ key: "k-1", now })).toBeNull(); + expect((await storage.idempotencyStore.get({ key: "k-2", now }))?.key).toBe("k-2"); + }); + + it("names a resealed command's person under the key it moved to, keeping who else it named", async () => { + const { storage, reseal, opened } = await setup(); + const stored = await storage.personalKeys.add("c-1", "default", await newStoredKey()); + const nickname = await seal({ + key: await openedKey(stored), + person: "c-1", + category: "default", + value: "Ada", + }); + await storage.scheduler.schedule({ + dedupeKey: "a", + command: { type: "NameCustomer", aggregateId: "c-1", payload: { nickname } }, + executeAt: new Date("2026-02-01T00:00:00.000Z"), + context: { correlationId: "c", causationId: "c", depth: 0 }, + persons: [ + { person: "c-1", fingerprint: stored.fingerprint }, + { person: "c-2", fingerprint: "f-other" }, + { person: "c-1", fingerprint: null }, + ], + }); + expect((await reseal.commands(await storage.scheduler.list())).resealed).toBe(1); + const [entry] = await storage.scheduler.list(); + const profile = await storage.personalKeys.get("c-1", "profile"); + expect(entry?.persons).toEqual([ + { person: "c-2", fingerprint: "f-other" }, + { person: "c-1", fingerprint: null }, + { person: "c-1", fingerprint: profile?.fingerprint }, + ]); + expect(await opened(fieldOf(entry?.command.payload, "nickname"))).toBe("Ada"); + }); + + it("leaves a value no key opens any more as it is", async () => { + const { storage, reseal, append } = await setup(); + const gone = await openedKey(await newStoredKey()); + const nickname = await seal({ key: gone, person: "c-1", category: "default", value: "Ada" }); + await append( + event("customer", "c-1", 1, "CustomerProfiled", { email: "ada@example.com", nickname }), + ); + await reseal.events(await all(storage)); + const [profiled] = await all(storage); + expect(fieldOf(profiled?.payload, "nickname")).toEqual(nickname); + expect((await storage.personalKeys.list("c-1")).map(({ category }) => category)).toEqual([ + "default", + ]); + }); +}); + describe("resealing a process's lifecycle events", () => { it("seals their state under whom it names, with no snapshot to drop", async () => { const { storage, reseal, append, opened } = await setup(); @@ -203,7 +342,7 @@ describe("resealing a process's lifecycle events", () => { expect(await reseal.events(await all(storage))).toMatchObject({ resealed: 1 }); const [handled] = await all(storage); expect(handled?.person).toBe("c-1"); - expect(await opened("c-1", fieldOf(fieldOf(handled?.payload, "state"), "email"))).toBe( + expect(await opened(fieldOf(fieldOf(handled?.payload, "state"), "email"))).toBe( "ada@example.com", ); }); @@ -237,15 +376,15 @@ describe("resealing scheduled commands", () => { unsealableTypes: new Set(["ShipOrder"]), }); const [a, b, c, d] = await storage.scheduler.list(); - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", "default"); expect(a?.persons).toEqual([ { person: "c-1", fingerprint: old.fingerprint }, { person: "c-1", fingerprint: key?.fingerprint }, ]); - expect(await opened("c-1", fieldOf(a?.command.payload, "email"))).toBe("ada@example.com"); + expect(await opened(fieldOf(a?.command.payload, "email"))).toBe("ada@example.com"); expect(b?.command.payload).toEqual({ customerId: "", address: "Bilbao" }); expect(c?.persons).toHaveLength(1); - expect(await storage.personalKeys.get("c-9")).toBeNull(); + expect(await storage.personalKeys.get("c-9", "default")).toBeNull(); expect(d?.command.payload).toEqual({ email: "x@example.com" }); }); }); @@ -278,7 +417,7 @@ describe("resealing dead letters", () => { event("customer", "c-1", 1, "CustomerRegistered", { email: "ada@example.com" }), event("customer", "c-3", 1, "CustomerRegistered", {}), ); - await storage.personalKeys.add("c-1", await newStoredKey()); + await storage.personalKeys.add("c-1", "default", await newStoredKey()); await storage.deadLetterStore.add(letter("policy")); await storage.deadLetterStore.add( letter("scheduled", { @@ -314,18 +453,18 @@ describe("resealing dead letters", () => { }); const policy = await storage.deadLetterStore.get("policy"); expect(policy?.person).toBe("c-1"); - expect(await opened("c-1", policy?.errorMessage)).toBe("cannot greet ada@example.com"); - expect(await opened("c-1", policy?.errorStack)).toContain("at greet"); + expect(await opened(policy?.errorMessage)).toBe("cannot greet ada@example.com"); + expect(await opened(policy?.errorStack)).toContain("at greet"); expect(policy).not.toHaveProperty("payload"); expect(policy).not.toHaveProperty("persons"); const scheduled = await storage.deadLetterStore.get("scheduled"); - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", "default"); expect(scheduled).toMatchObject({ person: "c-1", persons: [{ person: "c-1", fingerprint: key?.fingerprint }], }); - expect(await opened("c-1", fieldOf(scheduled?.payload, "email"))).toBe("ada@example.com"); - expect(await opened("c-1", scheduled?.errorMessage)).toBe("cannot greet ada@example.com"); + expect(await opened(fieldOf(scheduled?.payload, "email"))).toBe("ada@example.com"); + expect(await opened(scheduled?.errorMessage)).toBe("cannot greet ada@example.com"); for (const id of ["since", "nobody", "missing", "named", "nameless"]) { expect(await storage.deadLetterStore.get(id)).toMatchObject({ errorMessage: "cannot greet ada@example.com", @@ -357,15 +496,31 @@ describe("resealing dead letters", () => { resealed: 2, discarded: 2, }); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).toBeNull(); const policy = await storage.deadLetterStore.get("policy"); expect(isSealed(policy?.errorMessage)).toBe(true); expect(policy).not.toHaveProperty("errorStack"); }); + it("seals the text of a letter under every key the person holds, one inside the other", async () => { + const { storage, reseal, append, opened } = await setup(); + await append(event("customer", "c-1", 1, "CustomerRegistered", { email: "ada@example.com" })); + await storage.personalKeys.add("c-1", "default", await newStoredKey()); + const profile = await storage.personalKeys.add("c-1", "profile", await newStoredKey()); + await storage.deadLetterStore.add(letter("policy")); + await reseal.letters(await storage.deadLetterStore.list(), started); + const policy = await storage.deadLetterStore.get("policy"); + expect(policy?.errorMessage).toMatchObject({ + $personal: { person: "c-1", category: "profile", key: profile.fingerprint }, + }); + expect(await opened(policy?.errorMessage)).toBe("cannot greet ada@example.com"); + await storage.personalKeys.remove("c-1", profile.fingerprint); + expect(await opened(policy?.errorMessage)).toBeNull(); + }); + it("seals the payload of a letter that names its person already, leaving its text as it is", async () => { const { storage, reseal, opened } = await setup(); - const key = await storage.personalKeys.add("c-1", await newStoredKey()); + const key = await storage.personalKeys.add("c-1", "default", await newStoredKey()); const sealedText = { $personal: { person: "c-1", key: key.fingerprint, iv: "iv", data: "d" } }; await storage.deadLetterStore.add( letter("scheduled", { @@ -395,7 +550,7 @@ describe("resealing dead letters", () => { errorMessage: sealedText, errorStack: sealedText, }); - expect(await opened("c-1", fieldOf(resealed?.payload, "email"))).toBe("ada@example.com"); + expect(await opened(fieldOf(resealed?.payload, "email"))).toBe("ada@example.com"); expect((await storage.deadLetterStore.get("unmarked"))?.payload).toEqual({ email: "ada@example.com", }); @@ -404,7 +559,7 @@ describe("resealing dead letters", () => { it("counts a letter discarded when its payload is, though its text opens under the key the person holds again", async () => { const { storage, reseal, append, opened } = await setup(); await append(forgotten("c-1")); - await storage.personalKeys.add("c-1", await newStoredKey()); + await storage.personalKeys.add("c-1", "default", await newStoredKey()); await storage.deadLetterStore.add( letter("scheduled", { kind: "scheduled", @@ -420,7 +575,7 @@ describe("resealing dead letters", () => { discarded: 1, }); const resealed = await storage.deadLetterStore.get("scheduled"); - expect(await opened("c-1", fieldOf(resealed?.payload, "email"))).toBeNull(); - expect(await opened("c-1", resealed?.errorMessage)).toBe("cannot greet ada@example.com"); + expect(await opened(fieldOf(resealed?.payload, "email"))).toBeNull(); + expect(await opened(resealed?.errorMessage)).toBe("cannot greet ada@example.com"); }); }); diff --git a/packages/core/src/kernel/personal-data/reseal-batches.ts b/packages/core/src/kernel/personal-data/reseal-batches.ts index 83405d52..922d0a84 100644 --- a/packages/core/src/kernel/personal-data/reseal-batches.ts +++ b/packages/core/src/kernel/personal-data/reseal-batches.ts @@ -5,9 +5,12 @@ import type { ScheduledCommand } from "../../adapter/storage/scheduler.ts"; import type { StoredEvent } from "../../contracts/event.ts"; import type { AggregatesRuntime } from "../aggregate/runtime.ts"; import type { ProcessesRuntime } from "../process/build-processes.ts"; -import { seal } from "./crypto.ts"; +import { isRecord } from "../shared/merge-fields.ts"; +import { isSealed, type PersonalKey } from "./crypto.ts"; +import { sealUnderAll } from "./errors.ts"; import { personOfEvent } from "./persons.ts"; import { + type ChosenKey, createResealKeys, planEventReseal, planPayloadReseal, @@ -104,13 +107,83 @@ const qualified = (event: StoredEvent): string => `${event.aggregateType}.${even const mergedPersons = ( persons: readonly KeyedPerson[] | undefined, - added: KeyedPerson, + added: readonly KeyedPerson[], ): readonly KeyedPerson[] => { const merged = new Map((persons ?? []).map((held) => [keyedPersonId(held), held])); - merged.set(keyedPersonId(added), added); + for (const held of added) merged.set(keyedPersonId(held), held); return [...merged.values()]; }; +interface Chosen { + readonly keys: ReadonlyMap; + readonly discarded: boolean; +} + +// The key of each category a plan seals, chosen by `choose`. +const chosenFor = async ( + categories: readonly string[], + choose: (category: string) => Promise, +): Promise => { + const keys = new Map(); + let discarded = false; + for (const category of categories) { + const chosen = await choose(category); + keys.set(category, chosen.key); + discarded ||= chosen.discarded; + } + return { keys, discarded }; +}; + +const keyOfChosen = + ({ keys }: Chosen) => + (category: string): PersonalKey => { + const key = keys.get(category); + if (key === undefined) throw new Error(`No key was chosen for the category "${category}"`); + return key; + }; + +// Whom a resealed payload names: who it named before, and the person under the keys its values +// are sealed under now, but for a key a moved value left that none of its values is under any more. +const personsAfter = ( + persons: readonly KeyedPerson[] | undefined, + person: string, + payload: unknown, + movedFrom: readonly string[], +): readonly KeyedPerson[] => { + const sealedUnder = new Set( + isRecord(payload) + ? Object.values(payload) + .filter(isSealed) + .filter(({ $personal }) => $personal.person === person) + .map(({ $personal }) => $personal.key) + : [], + ); + return mergedPersons( + (persons ?? []).filter( + (held) => + held.person !== person || + held.fingerprint === null || + !movedFrom.includes(held.fingerprint) || + sealedUnder.has(held.fingerprint), + ), + [...sealedUnder].map((fingerprint) => ({ person, fingerprint })), + ); +}; + +// A remembered rejection may quote a value read under the key it moved from, which forgetting its +// new category would not reach: it is decided again instead. +const dropCachedUnder = async ( + transaction: StorageTransaction, + person: string, + movedFrom: ReadonlySet, +): Promise => { + if (movedFrom.size === 0) return; + const keep = (await transaction.personalKeys.list(person)) + .map(({ key }) => key.fingerprint) + .filter((fingerprint) => !movedFrom.has(fingerprint)); + await transaction.idempotencyStore.forget(person, keep); +}; + const unsealable = (count: number, types: Iterable): Sealed => ({ ...NOTHING_SEALED, unsealable: count, @@ -144,7 +217,7 @@ export const createResealBatches: CreateResealBatchesFunction = ({ } let sealed = NOTHING_SEALED; for (const [person, items] of byPerson) { - await outside.get(person); + await outside.list(person); sealed = addSealed( sealed, await storage.transact((transaction) => @@ -171,12 +244,18 @@ export const createResealBatches: CreateResealBatchesFunction = ({ const discardedTypes = new Set(); const resealed = []; const streams = new Map(); + const moved = new Set(); for (const { event, plan } of items) { - const chosen = await keys.event(plan.person, event.position); + const opened = await plan.opened(keys.open); + if (opened === null) continue; + for (const fingerprint of opened.movedFrom) moved.add(fingerprint); + const chosen = await chosenFor(opened.categories, (category) => + keys.event(plan.person, category, event.position), + ); if (chosen.discarded) discardedTypes.add(qualified(event)); resealed.push({ position: event.position, - payload: await plan.seal(chosen.key), + payload: await opened.seal(keyOfChosen(chosen)), person: plan.person, discarded: chosen.discarded, }); @@ -193,6 +272,8 @@ export const createResealBatches: CreateResealBatchesFunction = ({ await transaction.eventStore.reseal( resealed.map(({ position, payload, person }) => ({ position, payload, person })), ); + const [first] = items; + if (first !== undefined) await dropCachedUnder(transaction, first.plan.person, moved); // A snapshot folded from what was in the clear names nobody a forget would remove it for. for (const { aggregateType, aggregateId } of streams.values()) { if ((aggregates.byName[aggregateType]?.snapshot ?? null) === null) continue; @@ -223,20 +304,24 @@ export const createResealBatches: CreateResealBatchesFunction = ({ } const sealed = await perPerson(owned, async (items, transaction, keys) => { let discarded = 0; + let resealed = 0; for (const { entry, plan } of items) { - const chosen = await keys.payload(plan.person, null); + const opened = await plan.opened(keys.open); + if (opened === null) continue; + const chosen = await chosenFor(opened.categories, (category) => + keys.payload(plan.person, category, null), + ); if (chosen.discarded) discarded += 1; + const payload = await opened.seal(keyOfChosen(chosen)); await transaction.scheduler.reseal({ dedupeKey: entry.dedupeKey, from: entry.command.payload, - payload: await plan.seal(chosen.key), - persons: mergedPersons(entry.persons, { - person: plan.person, - fingerprint: chosen.key.fingerprint, - }), + payload, + persons: personsAfter(entry.persons, plan.person, payload, opened.movedFrom), }); + resealed += 1; } - return { ...NOTHING_SEALED, resealed: items.length, discarded }; + return { ...NOTHING_SEALED, resealed, discarded }; }); return addSealed( sealed, @@ -292,23 +377,28 @@ export const createResealBatches: CreateResealBatchesFunction = ({ let { person, errorMessage, errorStack, payload: stored, persons } = letter; let discarded = false; if (payload !== null) { - const chosen = await keys.payload(payload.person, letter.firstFailedAt); - discarded ||= chosen.discarded; - stored = await payload.seal(chosen.key); - persons = mergedPersons(persons, { - person: payload.person, - fingerprint: chosen.key.fingerprint, - }); + const opened = await payload.opened(keys.open); + if (opened !== null) { + const chosen = await chosenFor(opened.categories, (category) => + keys.payload(payload.person, category, letter.firstFailedAt), + ); + discarded ||= chosen.discarded; + stored = await opened.seal(keyOfChosen(chosen)); + persons = personsAfter(persons, payload.person, stored, opened.movedFrom); + } } if (text) { - const { key, discarded: thrownAway } = await keys.errorText(named, letter.lastFailedAt); + const { keys: under, discarded: thrownAway } = await keys.errorText( + named, + letter.lastFailedAt, + ); discarded ||= thrownAway; person = named; if (typeof errorMessage === "string") { - errorMessage = await seal({ key, person: named, value: errorMessage }); + errorMessage = await sealUnderAll(under, named, errorMessage); } if (typeof errorStack === "string") { - errorStack = await seal({ key, person: named, value: errorStack }); + errorStack = await sealUnderAll(under, named, errorStack); } } await transaction.deadLetterStore.reseal(letter.id, { diff --git a/packages/core/src/kernel/personal-data/resealing.test.ts b/packages/core/src/kernel/personal-data/resealing.test.ts index 5f999151..b1683c60 100644 --- a/packages/core/src/kernel/personal-data/resealing.test.ts +++ b/packages/core/src/kernel/personal-data/resealing.test.ts @@ -61,37 +61,58 @@ const event = (overrides: Partial): StoredEvent => ({ ...overrides, }); -const byAggregate: PersonalMarks = { person: null, fields: ["email"] }; -const byField: PersonalMarks = { person: "customerId", fields: ["address"] }; +const byAggregate: PersonalMarks = { person: null, fields: ["email"], categories: {} }; +const byField: PersonalMarks = { person: "customerId", fields: ["address"], categories: {} }; const sealedWith = async (plan: ResealPlanOf) => { if (plan === null || plan === "unsealable") throw new Error(`no plan: ${String(plan)}`); const key = await openedKey(await newStoredKey()); - return { person: plan.person, key, value: await plan.seal(key) }; + const opened = await plan.opened(async () => null); + if (opened === null) throw new Error("nothing to seal"); + return { + person: plan.person, + categories: opened.categories, + key, + value: await opened.seal(() => key), + }; }; describe("marksFingerprint", () => { it("is null when nothing is marked, and names each set of marks apart, whatever their order", () => { expect(marksFingerprint(runtimes({ events: { CustomerRegistered: null } }))).toBeNull(); const one = marksFingerprint( - runtimes({ events: { CustomerRegistered: { person: null, fields: ["email", "name"] } } }), + runtimes({ + events: { CustomerRegistered: { person: null, fields: ["email", "name"], categories: {} } }, + }), ); expect(one).toEqual( marksFingerprint( - runtimes({ events: { CustomerRegistered: { person: null, fields: ["name", "email"] } } }), + runtimes({ + events: { + CustomerRegistered: { person: null, fields: ["name", "email"], categories: {} }, + }, + }), ), ); const others = [ - runtimes({ events: { CustomerRegistered: { person: null, fields: ["email"] } } }), - runtimes({ events: { CustomerRegistered: { person: "email", fields: ["name"] } } }), - runtimes({ commands: { CustomerRegistered: { person: null, fields: ["email", "name"] } } }), - runtimes({ process: { person: null, fields: ["email", "name"] } }), + runtimes({ + events: { CustomerRegistered: { person: null, fields: ["email"], categories: {} } }, + }), + runtimes({ + events: { CustomerRegistered: { person: "email", fields: ["name"], categories: {} } }, + }), + runtimes({ + commands: { + CustomerRegistered: { person: null, fields: ["email", "name"], categories: {} }, + }, + }), + runtimes({ process: { person: null, fields: ["email", "name"], categories: {} } }), ].map(marksFingerprint); expect(new Set([one, ...others]).size).toBe(5); }); it("names what each mark belongs to, and is the same whatever order the registry lists them in", () => { - const marks = { person: null, fields: ["email"] }; + const marks = { person: null, fields: ["email"], categories: {} }; const fingerprintOf = (marked: Marked) => marksFingerprint(runtimes(marked)); expect(fingerprintOf({ events: { A: marks, B: null } })).not.toEqual( fingerprintOf({ events: { A: null, B: marks } }), @@ -99,8 +120,10 @@ describe("marksFingerprint", () => { expect(fingerprintOf({ commands: { A: marks, B: null } })).not.toEqual( fingerprintOf({ commands: { A: null, B: marks } }), ); - expect(fingerprintOf({ events: { A: marks, B: { person: "x", fields: [] } } })).toEqual( - fingerprintOf({ events: { B: { person: "x", fields: [] }, A: marks } }), + expect( + fingerprintOf({ events: { A: marks, B: { person: "x", fields: [], categories: {} } } }), + ).toEqual( + fingerprintOf({ events: { B: { person: "x", fields: [], categories: {} }, A: marks } }), ); const named = (name: string) => marksFingerprint({ @@ -109,12 +132,24 @@ describe("marksFingerprint", () => { }); expect(named("a.one")).not.toEqual(named("a.two")); }); + + it("names a field of the default category as before categories, and one of another category apart", () => { + const fingerprintOf = (categories: Readonly>) => + marksFingerprint( + runtimes({ + events: { CustomerRegistered: { person: null, fields: ["email"], categories } }, + }), + ); + expect(fingerprintOf({ email: "default" })).toEqual(fingerprintOf({})); + expect(fingerprintOf({ email: "profile" })).not.toEqual(fingerprintOf({})); + expect(fingerprintOf({ email: "profile" })).not.toEqual(fingerprintOf({ email: "billing" })); + }); }); describe("planEventReseal", () => { const { aggregates, processes } = runtimes({ events: { CustomerRegistered: byAggregate, OrderPlaced: byField, CustomerNoted: null }, - process: { person: "customerId", fields: ["email"] }, + process: { person: "customerId", fields: ["email"], categories: {} }, }); const plan = (overrides: Partial) => planEventReseal({ event: event(overrides), aggregates, processes }); @@ -135,13 +170,46 @@ describe("planEventReseal", () => { expect(ordered.person).toBe("7"); }); + it("names the categories of the values in the clear, and seals each under its category's key", async () => { + const marks: PersonalMarks = { + person: null, + fields: ["email", "phone", "address"], + categories: { phone: "profile", address: "profile" }, + }; + const planned = planEventReseal({ + event: event({ payload: { email: "a", phone: "1", address: null } }), + aggregates: runtimes({ events: { CustomerRegistered: marks } }).aggregates, + processes, + }); + if (planned === null || planned === "unsealable") throw new Error("no plan"); + const opened = await planned.opened(async () => null); + expect(opened?.categories).toEqual(["default", "profile"]); + const keys = { + default: await openedKey(await newStoredKey()), + profile: await openedKey(await newStoredKey()), + }; + const value = (await opened?.seal((category) => keys[category as keyof typeof keys])) as Record< + string, + { $personal: { category?: string; key: string } } + >; + expect(value.email?.$personal).not.toHaveProperty("category"); + expect(value.email?.$personal.key).toBe(keys.default.fingerprint); + expect(value.phone?.$personal).toMatchObject({ + category: "profile", + key: keys.profile.fingerprint, + }); + expect(await unseal({ key: keys.profile, sealed: value.phone as never })).toBe("1"); + }); + it("keeps what is sealed already and seals the rest, and seals a null as the event store does", async () => { const key = await openedKey(await newStoredKey()); - const address = await seal({ key, person: "c-2", value: "Bilbao" }); + const address = await seal({ key, person: "c-2", category: "default", value: "Bilbao" }); const partly = planEventReseal({ event: event({ payload: { email: null }, person: null }), aggregates: runtimes({ - events: { CustomerRegistered: { person: null, fields: ["email", "address"] } }, + events: { + CustomerRegistered: { person: null, fields: ["email", "address"], categories: {} }, + }, }).aggregates, processes, }); @@ -151,7 +219,9 @@ describe("planEventReseal", () => { planEventReseal({ event: event({ payload: { email: "a", address }, person: "c-1" }), aggregates: runtimes({ - events: { CustomerRegistered: { person: null, fields: ["email", "address"] } }, + events: { + CustomerRegistered: { person: null, fields: ["email", "address"], categories: {} }, + }, }).aggregates, processes, }), @@ -226,7 +296,7 @@ describe("planEventReseal", () => { processes, }), ).toBeNull(); - const byInstance = runtimes({ process: { person: null, fields: ["email"] } }); + const byInstance = runtimes({ process: { person: null, fields: ["email"], categories: {} } }); const instance = await sealedWith( planEventReseal({ event: event({ @@ -257,7 +327,16 @@ describe("planPayloadReseal", () => { const byId = await sealedWith( planPayloadReseal({ marks: byAggregate, payload: { email: "a" }, aggregateId: "c-1" }), ); - expect(byId.person).toBe("c-1"); + expect(byId).toMatchObject({ person: "c-1", categories: ["default"] }); + const categorized = await sealedWith( + planPayloadReseal({ + marks: { ...byAggregate, categories: { email: "profile" } }, + payload: { email: "a" }, + aggregateId: "c-1", + }), + ); + expect(categorized.categories).toEqual(["profile"]); + expect(categorized.value.email).toMatchObject({ $personal: { category: "profile" } }); }); it("has nothing to seal without a value in the clear, and nobody to seal under without a person", () => { @@ -273,13 +352,13 @@ describe("planPayloadReseal", () => { }); describe("createResealKeys", () => { - const forgotten = (person: string, version: number) => ({ + const forgotten = (person: string, version: number, payload: object = {}) => ({ id: `forgotten-${person}-${version}`, aggregateType: BOUNDA_AGGREGATE, aggregateId: person, version, type: PERSONAL_DATA_FORGOTTEN_EVENT, - payload: { personId: person }, + payload: { personId: person, ...payload }, timestamp: "2026-01-02T00:00:00.000Z", metadata: { correlationId: "c", @@ -309,14 +388,14 @@ describe("createResealKeys", () => { ], }); } - // Forgotten at position 4, after the events at 1 to 3. + // Forgotten at position 4, after the events at 1 to 3, by a forgetting from before categories. await storage.eventStore.append({ aggregateType: BOUNDA_AGGREGATE, aggregateId: "gone", expectedVersion: 0, events: [forgotten("gone", 1)], }); - const back = await storage.personalKeys.add("back", await newStoredKey()); + const back = await storage.personalKeys.add("back", "default", await newStoredKey()); await storage.eventStore.append({ aggregateType: BOUNDA_AGGREGATE, aggregateId: "back", @@ -330,7 +409,7 @@ describe("createResealKeys", () => { expectedVersion: 1, events: [{ ...forgotten("back", 2), type: "ProcessingRestricted", id: "restricted-back" }], }); - const kept = await storage.personalKeys.add("kept", await newStoredKey()); + const kept = await storage.personalKeys.add("kept", "default", await newStoredKey()); const keys = createResealKeys({ transaction: storage, keys: keysOver(storage.personalKeys) }); return { storage, keys, back, kept }; }; @@ -340,69 +419,186 @@ describe("createResealKeys", () => { const reads: string[] = []; const counted = keysOver({ ...storage.personalKeys, - get: (person: string) => { - reads.push(person); - return storage.personalKeys.get(person); + get: (person: string, category: string) => { + reads.push(`get ${person}`); + return storage.personalKeys.get(person, category); + }, + list: (person: string) => { + reads.push(`list ${person}`); + return storage.personalKeys.list(person); }, }); const keys = createResealKeys({ transaction: storage, keys: counted }); for (const at of [null, "2026-01-01T00:00:00.000Z"]) { - expect(await keys.payload("kept", at)).toMatchObject({ + expect(await keys.payload("kept", "default", at)).toMatchObject({ key: { fingerprint: kept.fingerprint }, }); } expect(await keys.errorText("kept", "2026-01-01T00:00:00.000Z")).toMatchObject({ discarded: false, }); - expect(reads).toEqual(["kept"]); + expect(reads).toEqual(["get kept", "list kept"]); }); it("discards a key for what an event held of a person forgotten after it, and creates one otherwise", async () => { const { storage, keys, back } = await setup(); - expect(await keys.event("gone", 3)).toMatchObject({ discarded: true }); - expect(await storage.personalKeys.get("gone")).toBeNull(); - const again = await keys.event("gone", 3); - expect(again.key).toBe((await keys.event("gone", 1)).key); - expect(await keys.event("back", 4)).toMatchObject({ discarded: true }); - expect(await keys.event("back", 5)).toMatchObject({ discarded: false }); - expect(await keys.event("back", 6)).toMatchObject({ + expect(await keys.event("gone", "default", 3)).toMatchObject({ discarded: true }); + expect(await storage.personalKeys.list("gone")).toEqual([]); + const again = await keys.event("gone", "default", 3); + expect(again.key).toBe((await keys.event("gone", "default", 1)).key); + expect(await keys.event("back", "default", 4)).toMatchObject({ discarded: true }); + expect(await keys.event("back", "default", 6)).toMatchObject({ discarded: false, key: { fingerprint: back.fingerprint }, }); - const fresh = await keys.event("new", 1); + const fresh = await keys.event("new", "default", 1); expect(fresh.discarded).toBe(false); - expect((await storage.personalKeys.get("new"))?.fingerprint).toBe(fresh.key.fingerprint); - expect((await keys.event("new", 2)).key).toBe(fresh.key); + expect((await storage.personalKeys.get("new", "default"))?.fingerprint).toBe( + fresh.key.fingerprint, + ); + expect((await keys.event("new", "default", 2)).key.fingerprint).toBe(fresh.key.fingerprint); + }); + + it("discards a key only for the categories a forgetting covered, and reads what it retained under the key it kept", async () => { + const { storage } = await setup(); + const held = await storage.personalKeys.add("ana", "default", await newStoredKey()); + const invoices = await storage.personalKeys.add("ana", "invoices", await newStoredKey()); + await storage.personalKeys.retain("ana", invoices.fingerprint, { + since: "2026-01-02T00:00:00.000Z", + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + await storage.eventStore.append({ + aggregateType: BOUNDA_AGGREGATE, + aggregateId: "ana", + expectedVersion: 0, + events: [ + forgotten("ana", 1, { + whole: false, + categories: ["profile"], + retained: ["invoices"], + expired: [], + }), + ], + }); + const keys = createResealKeys({ transaction: storage, keys: keysOver(storage.personalKeys) }); + expect(await keys.event("ana", "profile", 3)).toMatchObject({ discarded: true }); + expect(await keys.event("ana", "default", 3)).toMatchObject({ + discarded: false, + key: { fingerprint: held.fingerprint }, + }); + expect(await keys.event("ana", "invoices", 3)).toMatchObject({ + discarded: false, + key: { fingerprint: invoices.fingerprint }, + }); + expect(await keys.event("ana", "profile", 9)).toMatchObject({ discarded: false }); + expect(await storage.personalKeys.get("ana", "profile")).not.toBeNull(); + expect(await keys.payload("ana", "invoices", "2026-01-01T00:00:00.000Z")).toMatchObject({ + discarded: true, + }); + }); + + it("reads what a forgetting retained under the latest key it kept of that category, and discards once none is kept", async () => { + const { storage } = await setup(); + const kept = async (category: string, since: string) => { + const key = await storage.personalKeys.add("bo", category, await newStoredKey()); + await storage.personalKeys.retain("bo", key.fingerprint, { + since, + lastActiveAt: "2025-12-01T00:00:00.000Z", + }); + return key; + }; + await kept("invoices", "2025-12-15T00:00:00.000Z"); + const latest = await kept("invoices", "2026-01-01T00:00:00.000Z"); + await kept("invoices", "2026-02-01T00:00:00.000Z"); + await kept("profile", "2026-01-01T12:00:00.000Z"); + const retained = { whole: true, categories: ["default"], retained: ["invoices"], expired: [] }; + await storage.eventStore.append({ + aggregateType: BOUNDA_AGGREGATE, + aggregateId: "bo", + expectedVersion: 0, + events: [forgotten("bo", 1, retained)], + }); + const keys = createResealKeys({ transaction: storage, keys: keysOver(storage.personalKeys) }); + expect(await keys.event("bo", "invoices", 3)).toMatchObject({ + discarded: false, + key: { fingerprint: latest.fingerprint, retained: true }, + }); + await storage.eventStore.append({ + aggregateType: BOUNDA_AGGREGATE, + aggregateId: "cy", + expectedVersion: 0, + events: [forgotten("cy", 1, retained)], + }); + expect(await keys.event("cy", "invoices", 3)).toMatchObject({ discarded: true }); + }); + + it("goes by the last forgetting that covered the category", async () => { + const { storage } = await setup(); + const held = await storage.personalKeys.add("di", "invoices", await newStoredKey()); + await storage.personalKeys.retain("di", held.fingerprint, { + since: "2026-01-02T00:00:00.000Z", + lastActiveAt: "2026-01-01T00:00:00.000Z", + }); + await storage.eventStore.append({ + aggregateType: BOUNDA_AGGREGATE, + aggregateId: "di", + expectedVersion: 0, + events: [ + forgotten("di", 1, { whole: false, categories: ["invoices"], retained: [], expired: [] }), + forgotten("di", 2, { whole: false, categories: [], retained: ["invoices"], expired: [] }), + ], + }); + const keys = createResealKeys({ transaction: storage, keys: keysOver(storage.personalKeys) }); + expect(await keys.event("di", "invoices", 3)).toMatchObject({ + discarded: false, + key: { fingerprint: held.fingerprint }, + }); + }); + + it("discards a key for what an event held of a category declared after the person was forgotten as a whole", async () => { + const { storage } = await setup(); + await storage.eventStore.append({ + aggregateType: BOUNDA_AGGREGATE, + aggregateId: "ana", + expectedVersion: 0, + events: [ + forgotten("ana", 1, { whole: true, categories: ["default"], retained: [], expired: [] }), + ], + }); + const keys = createResealKeys({ transaction: storage, keys: keysOver(storage.personalKeys) }); + expect(await keys.event("ana", "default", 3)).toMatchObject({ discarded: true }); + expect(await keys.event("ana", "profile", 3)).toMatchObject({ discarded: true }); + expect(await storage.personalKeys.list("ana")).toEqual([]); }); it("seals a payload under the key the person holds unless forgotten after it was filed, and creates none for a forgotten person", async () => { const { storage, keys, back, kept } = await setup(); - expect(await keys.payload("kept", null)).toMatchObject({ + expect(await keys.payload("kept", "default", null)).toMatchObject({ discarded: false, key: { fingerprint: kept.fingerprint }, }); - expect(await keys.payload("back", "2026-01-01T00:00:00.000Z")).toMatchObject({ + expect(await keys.payload("back", "default", "2026-01-01T00:00:00.000Z")).toMatchObject({ discarded: true, }); - expect(await keys.payload("back", "2026-01-03T00:00:00.000Z")).toMatchObject({ + expect(await keys.payload("back", "default", "2026-01-03T00:00:00.000Z")).toMatchObject({ discarded: false, key: { fingerprint: back.fingerprint }, }); - expect(await keys.payload("back", "2026-01-02T00:00:00.000Z")).toMatchObject({ + expect(await keys.payload("back", "default", "2026-01-02T00:00:00.000Z")).toMatchObject({ discarded: false, }); - expect(await keys.payload("back", null)).toMatchObject({ discarded: false }); - expect(await keys.payload("gone", null)).toMatchObject({ discarded: true }); - expect(await storage.personalKeys.get("gone")).toBeNull(); - expect(await keys.payload("new", null)).toMatchObject({ discarded: false }); - expect(await storage.personalKeys.get("new")).not.toBeNull(); + expect(await keys.payload("back", "default", null)).toMatchObject({ discarded: false }); + expect(await keys.payload("gone", "default", null)).toMatchObject({ discarded: true }); + expect(await storage.personalKeys.list("gone")).toEqual([]); + expect(await keys.payload("new", "default", null)).toMatchObject({ discarded: false }); + expect(await storage.personalKeys.get("new", "default")).not.toBeNull(); }); - it("seals the text of an error under the key the person holds, never a new one", async () => { + it("seals the text of an error under every key the person holds, never a new one", async () => { const { storage, keys, kept } = await setup(); expect(await keys.errorText("kept", "2026-01-01T00:00:00.000Z")).toMatchObject({ discarded: false, - key: { fingerprint: kept.fingerprint }, + keys: [{ category: "default", key: { fingerprint: kept.fingerprint } }], }); expect(await keys.errorText("back", "2026-01-01T00:00:00.000Z")).toMatchObject({ discarded: true, @@ -415,7 +611,30 @@ describe("createResealKeys", () => { }); expect(await keys.errorText("new", "2026-01-01T00:00:00.000Z")).toMatchObject({ discarded: true, + keys: [{ category: "default" }], + }); + expect(await storage.personalKeys.list("new")).toEqual([]); + }); + + it("seals the text of an error under the keys of every category in order, past a forgetting that only ended a retention", async () => { + const { storage } = await setup(); + const profile = await storage.personalKeys.add("ana", "profile", await newStoredKey()); + const held = await storage.personalKeys.add("ana", "default", await newStoredKey()); + await storage.eventStore.append({ + aggregateType: BOUNDA_AGGREGATE, + aggregateId: "ana", + expectedVersion: 0, + events: [ + forgotten("ana", 1, { whole: false, categories: [], retained: [], expired: ["invoices"] }), + ], + }); + const keys = createResealKeys({ transaction: storage, keys: keysOver(storage.personalKeys) }); + expect(await keys.errorText("ana", "2026-01-01T00:00:00.000Z")).toMatchObject({ + discarded: false, + keys: [ + { category: "default", key: { fingerprint: held.fingerprint } }, + { category: "profile", key: { fingerprint: profile.fingerprint } }, + ], }); - expect(await storage.personalKeys.get("new")).toBeNull(); }); }); diff --git a/packages/core/src/kernel/personal-data/resealing.ts b/packages/core/src/kernel/personal-data/resealing.ts index 2a22f9ea..f537a585 100644 --- a/packages/core/src/kernel/personal-data/resealing.ts +++ b/packages/core/src/kernel/personal-data/resealing.ts @@ -1,15 +1,30 @@ import type { StorageTransaction } from "../../adapter/adapter.ts"; +import { + DEFAULT_PERSONAL_CATEGORY, + type SealedValue, +} from "../../adapter/storage/personal-key-store.ts"; import type { StoredEvent } from "../../contracts/event.ts"; -import { BOUNDA_AGGREGATE, PERSONAL_DATA_FORGOTTEN_EVENT } from "../../modules/bounda-events.ts"; +import { + BOUNDA_AGGREGATE, + PERSONAL_DATA_FORGOTTEN_EVENT, + type PersonalDataForgottenPayload, +} from "../../modules/bounda-events.ts"; import type { AggregatesRuntime } from "../aggregate/runtime.ts"; import type { ProcessesRuntime } from "../process/build-processes.ts"; import { processOfStream } from "../process/lifecycle.ts"; import { digest } from "../read-model/fingerprint.ts"; import { isRecord } from "../shared/merge-fields.ts"; -import { createDisposableKey, isSealed, type PersonalKey, seal } from "./crypto.ts"; -import type { PersonalMarks } from "./marks.ts"; +import { + categoryOf, + createDisposableKey, + isSealed, + type PersonalKey, + seal, + unseal, +} from "./crypto.ts"; +import { fieldCategory, type PersonalMarks } from "./marks.ts"; import { lifecycleStateOf, sealState, statePersonOf } from "./process-state.ts"; -import type { PersonalKeys } from "./wrapped-keys.ts"; +import { type CategoryKey, inSealingOrder, type PersonalKeys } from "./wrapped-keys.ts"; export interface MarksFingerprintArgs { readonly aggregates: AggregatesRuntime; @@ -21,7 +36,19 @@ export interface MarksFingerprintFunction { } const entryOf = (name: string, marks: PersonalMarks | null): readonly string[] => - marks === null ? [] : [JSON.stringify([name, marks.person, [...marks.fields].sort()])]; + marks === null + ? [] + : [ + JSON.stringify([ + name, + marks.person, + // A field of the default category is named as it was before categories existed. + [...marks.fields].sort().map((field) => { + const category = fieldCategory(marks, field); + return category === DEFAULT_PERSONAL_CATEGORY ? field : [field, category]; + }), + ]), + ]; /** * Names what every event, command and process state of the registry marks, so any change to a @@ -52,27 +79,42 @@ export interface ChosenKey { readonly discarded: boolean; } +/** + * The keys to seal the text of an error with, one inside the other, as a failure seals it. + */ +export interface ChosenKeys { + readonly keys: readonly CategoryKey[]; + readonly discarded: boolean; +} + /** * Chooses, in the transaction that reseals, the key for what a record written in the clear holds * of a person, as the forget the record escaped would have left it. */ export interface ResealKeys { /** - * For an event at `position`: a key no store keeps when a forgetting of the person came after - * it, their key otherwise, created when they hold none. + * For a value of `category` in an event at `position`: a key no store keeps when a forgetting + * of that category of the person came after it, the key it retained when it retained the + * category instead, while it lasts, and their key in force otherwise, created when they hold + * none. + */ + event(person: string, category: string, position: number): Promise; + /** + * For a value of `category` in a stored command's payload, filed at `at` for a dead letter: + * their key in force when they hold one and that category was not forgotten after `at`, a new + * key when they hold none and it was never forgotten, and else a key no store keeps. */ - event(person: string, position: number): Promise; + payload(person: string, category: string, at: string | null): Promise; /** - * For a stored command's payload, filed at `at` for a dead letter: their key when they hold one - * and were not forgotten after `at`, a new key when they hold none and were never forgotten, - * and else a key no store keeps. + * For the text of an error filed at `at`: every key they hold when they hold one and nothing of + * theirs was forgotten after `at`, and else a key no store keeps, never a new one, as a failure + * seals it. */ - payload(person: string, at: string | null): Promise; + errorText(person: string, at: string): Promise; /** - * For the text of an error filed at `at`: their key when they hold one and were not forgotten - * after `at`, and else a key no store keeps, never a new one, as a failure seals it. + * Opens a value sealed under a key the person still holds, in force or kept, to seal it anew. */ - errorText(person: string, at: string): Promise; + open: OpenSealedFunction; } export interface CreateResealKeysArgs { @@ -93,41 +135,75 @@ export interface CreateResealKeysFunction { interface Forgetting { readonly position: number; readonly timestamp: string; + // `null` for one recorded before categories, which forgot every category. + readonly payload: PersonalDataForgottenPayload | null; } +// What a forgetting did to a category: forgot it, kept it, or nothing. An end of what an earlier +// one kept forgets only what that one kept, which the retention ends again on its own. +const coverOf = (forgetting: Forgetting, category: string): "forgot" | "retained" | null => { + const { payload } = forgetting; + if (payload === null || !Array.isArray(payload.categories)) return "forgot"; + const retained: readonly string[] = payload.retained; + const forgot: readonly string[] = payload.categories; + if (retained.includes(category)) return "retained"; + // A forgetting as a whole forgot every category it did not keep, one declared since included. + return payload.whole !== false || forgot.includes(category) ? "forgot" : null; +}; + const cached = () => { const values = new Map>(); - return (person: string, read: () => Promise): Promise => { - const known = values.get(person); + return (id: string, read: () => Promise): Promise => { + const known = values.get(id); if (known !== undefined) return known; const value = read(); - values.set(person, value); + values.set(id, value); return value; }; }; -// Each person's last forgetting, key and discarded key are read once per transaction, which a -// store whose transactions run one at a time keeps from changing while it runs. +// Each person's forgettings, keys and discarded key are read once per transaction, which a store +// whose transactions run one at a time keeps from changing while it runs. export const createResealKeys: CreateResealKeysFunction = ({ transaction, keys }) => { - const forgettings = cached(); + const forgettings = cached(); const held = new Map>(); const discarded = cached(); - const lastForgetting = (person: string): Promise => + const forgettingsOf = (person: string): Promise => forgettings(person, async () => { const { events } = await transaction.eventStore.load({ aggregateType: BOUNDA_AGGREGATE, aggregateId: person, }); - const last = events.findLast((event) => event.type === PERSONAL_DATA_FORGOTTEN_EVENT); - return last === undefined ? null : { position: last.position, timestamp: last.timestamp }; + return events + .filter((event) => event.type === PERSONAL_DATA_FORGOTTEN_EVENT) + .map((event) => ({ + position: event.position, + timestamp: event.timestamp, + payload: isRecord(event.payload) + ? (event.payload as unknown as PersonalDataForgottenPayload) + : null, + })); }); - const current = (person: string): Promise => { - const known = held.get(person); + // The last forgetting that forgot or kept the category, and what it did. + const lastCovering = async ( + person: string, + category: string, + ): Promise<{ readonly forgetting: Forgetting; readonly cover: "forgot" | "retained" } | null> => { + for (const forgetting of [...(await forgettingsOf(person))].reverse()) { + const cover = coverOf(forgetting, category); + if (cover !== null) return { forgetting, cover }; + } + return null; + }; + + const current = (person: string, category: string): Promise => { + const id = JSON.stringify([person, category]); + const known = held.get(id); if (known !== undefined) return known; - const read = keys.held(person); - held.set(person, read); + const read = keys.held(person, category); + held.set(id, read); return read; }; @@ -137,44 +213,102 @@ export const createResealKeys: CreateResealKeysFunction = ({ transaction, keys } }); // The key in force, which the add answers, is the one the person holds from then on. - const kept = async (person: string): Promise => { - const key = await current(person); + const kept = async (person: string, category: string): Promise => { + const key = await current(person, category); if (key !== null) return { key, discarded: false }; - const added = keys.create(person); - held.set(person, added); + const added = keys.create(person, category); + held.set(JSON.stringify([person, category]), added); return { key: await added, discarded: false }; }; + // The key a forgetting kept for the category, while it lasts. + const retainedBy = async ( + person: string, + category: string, + forgetting: Forgetting, + ): Promise => { + const [found] = (await keys.list(person)) + .filter( + ({ category: of, retainedSince }) => + of === category && retainedSince !== null && retainedSince <= forgetting.timestamp, + ) + .sort((a, b) => ((a.retainedSince ?? "") < (b.retainedSince ?? "") ? 1 : -1)); + if (found === undefined) return thrownAway(person); + return { key: { ...(await keys.open(person, found.key)), retained: true }, discarded: false }; + }; + return { - event: async (person, position) => { - const forgotten = await lastForgetting(person); - return forgotten !== null && forgotten.position > position - ? thrownAway(person) - : kept(person); + open: async (sealed) => { + const key = await keys.opened(sealed.$personal.person, sealed.$personal.key); + return key === null ? null : { value: await unseal({ key, sealed }) }; }, - payload: async (person, at) => { - const [forgotten, key] = await Promise.all([lastForgetting(person), current(person)]); - if (key === null) return forgotten === null ? kept(person) : thrownAway(person); - return at !== null && forgotten !== null && forgotten.timestamp > at + event: async (person, category, position) => { + const last = await lastCovering(person, category); + if (last === null || last.forgetting.position < position) return kept(person, category); + return last.cover === "retained" + ? retainedBy(person, category, last.forgetting) + : thrownAway(person); + }, + payload: async (person, category, at) => { + const [last, key] = await Promise.all([ + lastCovering(person, category), + current(person, category), + ]); + if (key === null) return last === null ? kept(person, category) : thrownAway(person); + return at !== null && last !== null && last.forgetting.timestamp > at ? thrownAway(person) : { key, discarded: false }; }, errorText: async (person, at) => { - const [forgotten, key] = await Promise.all([lastForgetting(person), current(person)]); - return key === null || (forgotten !== null && forgotten.timestamp > at) - ? thrownAway(person) - : { key, discarded: false }; + const [all, every] = await Promise.all([forgettingsOf(person), keys.heldAll(person)]); + const since = all.some( + (forgetting) => + forgetting.timestamp > at && + (forgetting.payload === null || + !Array.isArray(forgetting.payload.categories) || + forgetting.payload.categories.length + forgetting.payload.retained.length > 0), + ); + if (every.length > 0 && !since) { + return { + keys: inSealingOrder(every), + discarded: false, + }; + } + const { key } = await thrownAway(person); + return { keys: [{ category: DEFAULT_PERSONAL_CATEGORY, key }], discarded: true }; }, }; }; /** - * What resealing one record takes: the person its marks name, and how its stored value reads once - * what it holds of them in the clear is sealed under a key. + * What resealing one record takes: the person its marks name, the categories of the values it + * holds in the clear, and how its stored value reads once they are sealed, each under the key + * chosen for its category. */ export interface ResealPlan { readonly person: string; - seal(key: PersonalKey): Promise; + /** + * Opens the values sealed under a category their field is no longer in; `null` when nothing is + * left to write, as when no key opens any of them and the record names its person already. + */ + opened(open: OpenSealedFunction): Promise | null>; +} + +/** + * A plan once what it moves is opened: the categories of the values it seals, the fingerprints of + * the keys the opened values leave, and the record as it reads once they are sealed. + */ +export interface OpenedReseal { + readonly categories: readonly string[]; + readonly movedFrom: readonly string[]; + seal(keyOf: (category: string) => PersonalKey): Promise; +} + +/** + * Opens a sealed value, `null` when no key the person holds opens it any more. + */ +export interface OpenSealedFunction { + (sealed: SealedValue): Promise<{ readonly value: unknown } | null>; } /** @@ -187,19 +321,72 @@ export type ResealPlanOf = ResealPlan | "unsealable" | null; const personIn = (value: unknown): string | null => (typeof value === "string" && value !== "") || typeof value === "number" ? String(value) : null; +// What is still sealed after the moved values were opened stays as it is: its key is gone. const sealFields = async ( payload: Readonly>, + marks: PersonalMarks, fields: readonly string[], person: string, - key: PersonalKey, + keyOf: (category: string) => PersonalKey, ): Promise> => { const sealed: Record = { ...payload }; for (const field of fields) { - sealed[field] = await seal({ key, person, value: payload[field] }); + if (isSealed(payload[field])) continue; + const category = fieldCategory(marks, field); + sealed[field] = await seal({ key: keyOf(category), person, category, value: payload[field] }); } return sealed; }; +const categoriesOf = (marks: PersonalMarks, fields: readonly string[]): readonly string[] => [ + ...new Set(fields.map((field) => fieldCategory(marks, field))), +]; + +// The fields that hold a value sealed for the person under another category than the field's, as +// one marked again in another category since: they are opened and sealed anew under their own. +const movedFields = ( + marks: PersonalMarks, + fields: Readonly>, + person: string | null, +): readonly string[] => + person === null + ? [] + : marks.fields.filter((field) => { + const value = fields[field]; + return ( + isSealed(value) && + value.$personal.person === person && + categoryOf(value) !== fieldCategory(marks, field) + ); + }); + +interface Opened { + readonly fields: Readonly>; + // The fields that hold a value in the clear now: those that did, and the moved ones opened. + readonly pending: readonly string[]; + readonly movedFrom: readonly string[]; +} + +const openMoved = async ( + fields: Readonly>, + clear: readonly string[], + moved: readonly string[], + open: OpenSealedFunction, +): Promise => { + const opened: Record = { ...fields }; + const pending = [...clear]; + const from = new Set(); + for (const field of moved) { + const sealed = fields[field] as SealedValue; + const value = await open(sealed); + if (value === null) continue; + opened[field] = value.value; + pending.push(field); + from.add(sealed.$personal.key); + } + return { fields: opened, pending, movedFrom: [...from] }; +}; + export interface PlanPayloadResealArgs { readonly marks: PersonalMarks | null; readonly payload: unknown; @@ -215,17 +402,30 @@ export interface PlanPayloadResealFunction { /** * For a command's payload, as a scheduled command or a dead letter holds it: its marked fields - * that hold a value in the clear, sealed as `sealCommandPayload` seals them. + * that hold a value in the clear, sealed as `sealCommandPayload` seals them, and those sealed under + * another category than the field's, sealed anew under their own. */ export const planPayloadReseal: PlanPayloadResealFunction = ({ marks, payload, aggregateId }) => { if (marks === null || !isRecord(payload)) return null; const clear = marks.fields.filter( (field) => payload[field] !== undefined && !isSealed(payload[field]), ); - if (clear.length === 0) return null; const person = marks.person === null ? aggregateId : personIn(payload[marks.person]); + const moved = movedFields(marks, payload, person); + if (clear.length === 0 && moved.length === 0) return null; if (person === null) return "unsealable"; - return { person, seal: (key) => sealFields(payload, clear, person, key) }; + return { + person, + opened: async (open) => { + const { fields, pending, movedFrom } = await openMoved(payload, clear, moved, open); + if (pending.length === 0) return null; + return { + categories: categoriesOf(marks, pending), + movedFrom, + seal: (keyOf) => sealFields(fields, marks, pending, person, keyOf), + }; + }, + }; }; export interface PlanEventResealArgs { @@ -241,18 +441,38 @@ export interface PlanEventResealFunction { const holdsValue = (value: unknown): boolean => value !== undefined && value !== null; // Clear values with nobody to seal them under, or under another person than the event names, are -// left as they are; an event with none is left alone whatever it names. +// left as they are; an event with none, and nothing moved, is left alone whatever it names. const planned = ( event: StoredEvent, + marks: PersonalMarks, + fields: Readonly>, clear: readonly string[], person: string | null, - seal: (person: string, key: PersonalKey) => Promise, + seal: ( + person: string, + fields: Readonly>, + pending: readonly string[], + keyOf: (category: string) => PersonalKey, + ) => Promise, ): ResealPlanOf => { - if (clear.length === 0 && event.person !== null) return null; + const moved = + event.person === null || event.person === person ? movedFields(marks, fields, person) : []; + if (clear.length === 0 && moved.length === 0 && event.person !== null) return null; if (person === null || (event.person !== null && event.person !== person)) { return clear.length === 0 ? null : "unsealable"; } - return { person, seal: (key) => seal(person, key) }; + return { + person, + opened: async (open) => { + const opened = await openMoved(fields, clear, moved, open); + if (opened.pending.length === 0 && event.person !== null) return null; + return { + categories: categoriesOf(marks, opened.pending), + movedFrom: opened.movedFrom, + seal: (keyOf) => seal(person, opened.fields, opened.pending, keyOf), + }; + }, + }; }; /** @@ -271,9 +491,14 @@ export const planEventReseal: PlanEventResealFunction = ({ event, aggregates, pr (field) => holdsValue(state[field]) && !isSealed(state[field]), ); const person = statePersonOf({ marks, state, instanceId: event.aggregateId }); - return planned(event, clear, person, async (named, key) => ({ + return planned(event, marks, state, clear, person, async (named, opened, _pending, keyOf) => ({ ...(event.payload as object), - state: await sealState({ marks, state, person: named, key: async () => key }), + state: await sealState({ + marks, + state: opened, + person: named, + key: async (_person, category) => keyOf(category), + }), })); } const marks = aggregates.byName[event.aggregateType]?.eventsByType[event.type]?.personal ?? null; @@ -283,5 +508,7 @@ export const planEventReseal: PlanEventResealFunction = ({ event, aggregates, pr (field) => payload[field] !== undefined && !isSealed(payload[field]), ); const person = marks.person === null ? event.aggregateId : personIn(payload[marks.person]); - return planned(event, clear, person, (named, key) => sealFields(payload, clear, named, key)); + return planned(event, marks, payload, clear, person, (named, opened, pending, keyOf) => + sealFields(opened, marks, pending, named, keyOf), + ); }; diff --git a/packages/core/src/kernel/personal-data/retention-app.test.ts b/packages/core/src/kernel/personal-data/retention-app.test.ts index 8059135f..34d0e8ce 100644 --- a/packages/core/src/kernel/personal-data/retention-app.test.ts +++ b/packages/core/src/kernel/personal-data/retention-app.test.ts @@ -98,13 +98,45 @@ const registry = { readModels: {}, } as unknown as Registry; +const categorized = { + aggregates: { + customer: { + state: { initialState: {} }, + events: { + customerTouched: { + payload: ({ z, personal }: EventPayloadArgs) => + z.object({ + note: personal(z.string()), + nickname: personal(z.string(), { category: "profile" as never }), + }), + evolve: () => ({}), + }, + }, + commands: { + touchCustomer: { + module: { + payload: ({ z }: PayloadArgs) => z.object({ customerId: z.string() }), + handler: ({ events }: { events: Record unknown> }) => [ + events.customerTouched?.({ note: "kept", nickname: "Al" }), + ], + }, + }, + }, + policies: {}, + processes: {}, + }, + }, + readModels: {}, + personalData: { categories: { profile: { basis: "consent", retention: "10d" } } }, +} as unknown as Registry; + const retention: Omit = { runtime: { personalData: { retention: "30d" } }, }; -const setUp = async (config = retention) => { +const setUp = async (config = retention, booted = registry) => { const adapter = memory(); - const { app, clock } = await createTestApp({ registry, config, adapter }); + const { app, clock } = await createTestApp({ registry: booted, config, adapter }); const storage = await adapter.createStorage(); const commands = app.commands as unknown as Record< string, @@ -146,8 +178,8 @@ describe("retention", () => { expect(await reasons()).toEqual([]); await advance(2); expect(await reasons()).toEqual([["c-1", "retention"]]); - expect(await storage.personalKeys.get("c-1")).toBeNull(); - expect(await storage.personalKeys.get("c-2")).not.toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).toBeNull(); + expect(await storage.personalKeys.get("c-2", "default")).not.toBeNull(); await advance(20); expect(await reasons()).toEqual([ ["c-1", "retention"], @@ -259,7 +291,27 @@ describe("retention", () => { expect(await rows()).toEqual(recorded); }); - it("runs no retention without the setting: nothing is recorded, due or forgotten", async () => { + it("forgets a declared category idle past its own retention, leaving the person's other data", async () => { + const { app, run, advance, forgotten, storage, clock } = await setUp({}, categorized); + const start = clock.now().getTime(); + await run("touchCustomer", "c-1"); + expect(await app.nextDueAt()).toEqual(new Date(start + 10 * DAY)); + await advance(11); + expect(await forgotten()).toEqual([ + expect.objectContaining({ + personId: "c-1", + reason: "retention", + whole: false, + categories: ["profile"], + }), + ]); + expect(await storage.personalKeys.get("c-1", "profile")).toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).not.toBeNull(); + expect((await storage.persons.get("c-1"))?.forgottenAt).toBeNull(); + expect(await app.nextDueAt()).toBeNull(); + }); + + it("runs no retention without the setting or a declared category: nothing is recorded, due or forgotten", async () => { const { app, run, advance, reasons, storage } = await setUp({}); await run("touchCustomer", "c-1"); expect(await app.nextDueAt()).toBeNull(); @@ -285,13 +337,17 @@ describe("forget in a reaction", () => { { personId: "c-1", reason: "requested", + whole: true, + categories: ["default"], + retained: [], + expired: [], fingerprints: [expect.any(String)], cancelledCommands: [], removedDeadLetters: [], }, ]); - expect(await storage.personalKeys.get("c-1")).toBeNull(); - expect(await storage.personalKeys.get("c-2")).not.toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).toBeNull(); + expect(await storage.personalKeys.get("c-2", "default")).not.toBeNull(); }); it("forgets once from a policy whose first attempt failed after forgetting", async () => { @@ -302,7 +358,7 @@ describe("forget in a reaction", () => { await run("requestErasure", "c-1"); expect(policyFailures).toBe(0); expect(await reasons()).toEqual([["c-1", "requested"]]); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).toBeNull(); expect(await app.deadLetters.list()).toEqual([]); const [event] = (await app.events.read({ limit: 1000 })).filter( (stored) => stored.type === "PersonalDataForgotten", diff --git a/packages/core/src/kernel/personal-data/retention.test.ts b/packages/core/src/kernel/personal-data/retention.test.ts index 03914ac9..6df48b70 100644 --- a/packages/core/src/kernel/personal-data/retention.test.ts +++ b/packages/core/src/kernel/personal-data/retention.test.ts @@ -4,14 +4,18 @@ import type { ForgetLedgerEntry } from "../../adapter/storage/forget-ledger.ts"; import type { StoredPerson } from "../../adapter/storage/person-store.ts"; import { testPersonalKey } from "../../adapter/testing/fixtures.ts"; import { createFixedClock, type FixedClock } from "../../contracts/clock.ts"; +import { retentionEnd } from "../../contracts/duration.ts"; import type { StoredEvent } from "../../contracts/event.ts"; import { createSequentialIdGenerator, type IdGenerator } from "../../contracts/ids.ts"; import { silentLogger } from "../../contracts/logger.ts"; import { memory } from "../../memory/index.ts"; import { createKeptPersonStore } from "../../memory/person-store.ts"; +import { createKeptPersonalKeyStore } from "../../memory/personal-key-store.ts"; +import type { PersonalDataModule } from "../../modules/personal-data.ts"; import { LEASE_MS } from "../shared/lease.ts"; import { tracer } from "../telemetry.ts"; import { createRecordingLogger, withoutLedger } from "../test-support.ts"; +import { buildPersonalCategories, type PersonalCategories } from "./categories.ts"; import { forgetPersonalData } from "./forget.ts"; import { boundForgetLedger } from "./forget-ledger.ts"; import { HOLD_MAX_MS, HOLD_MS } from "./hold.ts"; @@ -23,6 +27,7 @@ import { RETENTION_LEASE, RETENTION_SUBSCRIBER, RETENTION_SWEEP_LIMIT, + retains, retentionDueOf, } from "./retention.ts"; @@ -46,8 +51,13 @@ const event = ( const activity = (position: number, person: string | null) => event(position, { aggregateType: "customer", type: "CustomerTouched", person }); -const forgotten = (position: number, person: string) => - event(position, { aggregateType: "bounda", type: "PersonalDataForgotten", person }); +const forgotten = (position: number, person: string, whole = true) => + event(position, { + aggregateType: "bounda", + type: "PersonalDataForgotten", + person, + payload: { personId: person, whole }, + }); const restricted = (position: number, person: string) => event(position, { aggregateType: "bounda", type: "ProcessingRestricted", person }); @@ -55,10 +65,22 @@ const restricted = (position: number, person: string) => const day = (position: number): string => new Date(Date.UTC(2026, 0, 1) + position * DAY).toISOString(); +const categoriesOf = ( + retentionMs: number | null, + module?: PersonalDataModule, +): PersonalCategories => + buildPersonalCategories({ + module, + retention: retentionMs === null ? null : { months: 0, ms: retentionMs }, + }); + +const noKeys = { touch: async () => undefined }; + describe("createRetentionSubscriber", () => { - it("records each person's last activity and last forgetting, once per batch", async () => { + it("records each person's last activity and last forgetting as a whole, once per batch", async () => { const { store } = createKeptPersonStore(); const writes: string[] = []; + const touches: (readonly [string, string])[] = []; const subscriber = createRetentionSubscriber({ persons: { ...store, @@ -67,6 +89,11 @@ describe("createRetentionSubscriber", () => { await store.record(recorded); }, }, + keys: { + touch: async (person, lastActiveAt) => { + touches.push([person, lastActiveAt]); + }, + }, }); expect(subscriber).toMatchObject({ name: RETENTION_SUBSCRIBER, kind: "retention" }); const done = await subscriber.process( @@ -79,26 +106,52 @@ describe("createRetentionSubscriber", () => { activity(6, null), forgotten(7, "c-4"), restricted(8, "c-1"), + forgotten(9, "c-5", false), ], span, ); - expect(done).toBe(8); + expect(done).toBe(9); expect(writes.sort()).toEqual(["c-1", "c-2", "c-4"]); + expect(touches.sort()).toEqual([ + ["c-1", day(3)], + ["c-2", day(2)], + ]); expect(await store.get("c-1")).toEqual({ person: "c-1", restrictedSince: null, lastActiveAt: day(3), + position: 3, forgottenAt: null, hold: null, }); expect(await store.get("c-2")).toMatchObject({ lastActiveAt: day(2), forgottenAt: day(4) }); expect(await store.get("c-3")).toBeNull(); expect(await store.get("c-4")).toMatchObject({ lastActiveAt: null, forgottenAt: day(7) }); + expect(await store.get("c-5")).toBeNull(); + }); + + it("follows the last activity on each of the person's keys in force, never on one retained", async () => { + const { store } = createKeptPersonStore(); + const keys = createKeptPersonalKeyStore({ + restricted: () => false, + lastActiveAt: () => null, + }).store; + await keys.add("c-1", "default", testPersonalKey("1")); + await keys.add("c-1", "invoices", testPersonalKey("2")); + await keys.retain("c-1", "f-2", { since: day(0), lastActiveAt: day(0) }); + const subscriber = createRetentionSubscriber({ persons: store, keys }); + await subscriber.process([activity(1, "c-1"), activity(2, "c-1")], span); + expect( + (await keys.list("c-1")).map(({ category, lastActiveAt }) => [category, lastActiveAt]), + ).toEqual([ + ["default", day(2)], + ["invoices", day(0)], + ]); }); it("clears a forgetting with the activity after it, and keeps the activity before it", async () => { const { store } = createKeptPersonStore(); - const subscriber = createRetentionSubscriber({ persons: store }); + const subscriber = createRetentionSubscriber({ persons: store, keys: noKeys }); await subscriber.process([activity(1, "c-1"), forgotten(2, "c-1")], span); expect(await store.get("c-1")).toMatchObject({ lastActiveAt: day(1), forgottenAt: day(2) }); await subscriber.process([forgotten(3, "c-1"), activity(4, "c-1")], span); @@ -108,7 +161,7 @@ describe("createRetentionSubscriber", () => { it("leaves the same rows when its events are delivered again, or from the start", async () => { const events = [activity(1, "c-1"), forgotten(2, "c-1"), activity(3, "c-2")]; const { store } = createKeptPersonStore(); - const subscriber = createRetentionSubscriber({ persons: store }); + const subscriber = createRetentionSubscriber({ persons: store, keys: noKeys }); await subscriber.process(events.slice(0, 2), span); await subscriber.process(events.slice(2), span); const rows = async (): Promise => @@ -119,21 +172,27 @@ describe("createRetentionSubscriber", () => { expect(await rows()).toEqual(before); const rebuilt = createKeptPersonStore().store; - await createRetentionSubscriber({ persons: rebuilt }).process(events, span); + await createRetentionSubscriber({ persons: rebuilt, keys: noKeys }).process(events, span); expect(await Promise.all(["c-1", "c-2"].map((person) => rebuilt.get(person)))).toEqual(before); }); }); -const setUp = async (storage?: Storage) => { +interface SetUpOptions { + readonly retentionMs?: number | null; + readonly module?: PersonalDataModule; +} + +const setUp = async (storage?: Storage, { retentionMs = 30 * DAY, module }: SetUpOptions = {}) => { const live = storage ?? (await memory().createStorage()); const clock = createFixedClock(new Date(day(0))); const ids = createSequentialIdGenerator(); const { logger, entries } = createRecordingLogger(); + const categories = categoriesOf(retentionMs, module); const sweepOver = (storage: Storage) => createRetentionSweep({ storage, intentions: withoutLedger({ ids, clock }), - retentionMs: 30 * DAY, + categories, ids, clock, logger, @@ -141,7 +200,7 @@ const setUp = async (storage?: Storage) => { batchSize: 2, }); const sweep = sweepOver(live); - const subscriber = createRetentionSubscriber({ persons: live.persons }); + const subscriber = createRetentionSubscriber({ persons: live.persons, keys: live.personalKeys }); // What the dispatcher does: the batch, then the checkpoint past it. const record = async (): Promise => { const through = await live.checkpointStore.get(RETENTION_SUBSCRIBER); @@ -181,6 +240,10 @@ const setUp = async (storage?: Storage) => { ], }); }; + const forgettings = async (): Promise => + (await live.eventStore.readAll({ afterPosition: 0, limit: 1000 })).filter( + (stored) => stored.type === "PersonalDataForgotten", + ); const forgottenFor = async (): Promise => (await live.eventStore.readAll({ afterPosition: 0, limit: 1000 })) .filter((stored) => stored.type === "PersonalDataForgotten") @@ -194,6 +257,8 @@ const setUp = async (storage?: Storage) => { ids, logger, entries, + categories, + forgettings, sweep, sweepOver, record, @@ -238,7 +303,11 @@ describe("createRetentionSweep", () => { fields: { personId: "c-1", reason: "retention", - hadKey: false, + whole: true, + categories: ["default"], + retained: [], + expired: [], + destroyedKeys: 0, cancelledCommands: 0, removedDeadLetters: 0, }, @@ -250,7 +319,7 @@ describe("createRetentionSweep", () => { }); it("forgets nobody whose activity or forgetting past the subscriber's checkpoint it has not recorded", async () => { - const { sweep, record, touch, forgottenFor, advance, clock, storage, ids, logger } = + const { sweep, record, touch, forgottenFor, advance, clock, storage, ids, logger, categories } = await setUp(); for (const person of ["c-1", "c-2", "c-3", "c-4"]) await touch(person); await record(); @@ -263,6 +332,8 @@ describe("createRetentionSweep", () => { clock, logger, concurrencyRetries: 0, + categories, + batchSize: 2, personId: "c-2", }); // Restrictions are not activity: a page of them, a batch long, reads on to the next. @@ -287,12 +358,12 @@ describe("createRetentionSweep", () => { }); it("forgets a person once when two instances sweep at the same time", async () => { - const { sweep, record, touch, forgottenFor, advance, clock, storage, ids, logger } = + const { sweep, record, touch, forgottenFor, advance, clock, storage, ids, logger, categories } = await setUp(); const other = createRetentionSweep({ storage, intentions: withoutLedger({ ids, clock }), - retentionMs: 30 * DAY, + categories, ids, clock, logger, @@ -338,16 +409,19 @@ describe("createRetentionSweep", () => { ...transaction, personalKeys: { ...transaction.personalKeys, - remove: async (person) => { + remove: async (person, fingerprint) => { attempts.push(person); if (person === "c-1") throw new Error("disk full"); - return transaction.personalKeys.remove(person); + return transaction.personalKeys.remove(person, fingerprint); }, }, }), ), }); const failedAt = Date.parse(day(32)); + for (const person of ["c-1", "c-2", "c-3"]) { + await live.personalKeys.add(person, "default", testPersonalKey(person)); + } await touch("c-1"); clock.set(new Date(day(1))); await touch("c-2"); @@ -469,6 +543,134 @@ describe("createRetentionSweep", () => { expect(await forgottenFor()).toEqual([["c-1", "retention"]]); }); + it("leaves alone a key whose person was restricted, or active again, after the list was read", async () => { + const live = await memory().createStorage(); + let change: (() => Promise) | undefined; + const profile = { categories: { profile: { basis: "consent", retention: "30d" } } } as const; + const { sweep, record, touch, advance, clock, forgettings, entries } = await setUp( + { + ...live, + personalKeys: { + ...live.personalKeys, + listDue: async (args) => { + const listed = await live.personalKeys.listDue(args); + const next = change; + change = undefined; + await next?.(); + return listed; + }, + }, + }, + { retentionMs: null, module: profile }, + ); + await live.personalKeys.add("c-1", "profile", testPersonalKey("1")); + await touch("c-1"); + await record(); + advance(31); + change = () => live.persons.restrict("c-1", clock.now()).then(() => undefined); + expect(await sweep.sweep(clock.now())).toBe(0); + await live.persons.unrestrict("c-1"); + change = () => live.personalKeys.touch("c-1", clock.now().toISOString()); + expect(await sweep.sweep(clock.now())).toBe(0); + expect(await forgettings()).toEqual([]); + expect(entries.filter(({ level }) => level === "error")).toEqual([]); + expect((await live.personalKeys.list("c-1")).map(({ category }) => category)).toEqual([ + "profile", + ]); + }); + + it("waits for activity it has not recorded of a key in force, never of one a forgetting kept", async () => { + const live = await memory().createStorage(); + const legal = { + categories: { + invoices: { basis: "legal-obligation", retention: "30d" }, + profile: { basis: "consent", retention: "30d" }, + }, + } as const; + const { sweep, record, touch, advance, clock, storage } = await setUp(live, { + retentionMs: null, + module: legal, + }); + await live.personalKeys.add("c-1", "profile", testPersonalKey("1")); + await live.personalKeys.add("c-1", "invoices", testPersonalKey("2")); + await touch("c-1"); + await record(); + advance(10); + await live.personalKeys.retain("c-1", testPersonalKey("2").fingerprint, { + since: clock.now().toISOString(), + lastActiveAt: day(0), + }); + advance(21); + await touch("c-1"); + expect(await sweep.sweep(clock.now())).toBe(1); + expect((await storage.personalKeys.list("c-1")).map(({ category }) => category)).toEqual([ + "profile", + ]); + }); + + it("leaves a kept key a legal obligation keeps again, or whose person was restricted, after the list", async () => { + const live = await memory().createStorage(); + let change: (() => Promise) | undefined; + const { sweep, clock, forgettings, entries } = await setUp( + { + ...live, + personalKeys: { + ...live.personalKeys, + listRetained: async (args) => { + const listed = await live.personalKeys.listRetained(args); + const next = change; + change = undefined; + await next?.(); + return listed; + }, + }, + }, + { retentionMs: null, module: { categories: {} } }, + ); + await live.personalKeys.add("c-1", "orders", testPersonalKey("1")); + await live.personalKeys.retain("c-1", testPersonalKey("1").fingerprint, { + since: day(0), + lastActiveAt: day(0), + }); + change = () => live.persons.restrict("c-1", clock.now()).then(() => undefined); + expect(await sweep.sweep(clock.now())).toBe(0); + await live.persons.unrestrict("c-1"); + change = async () => { + await live.personalKeys.remove("c-1", testPersonalKey("1").fingerprint); + }; + expect(await sweep.sweep(clock.now())).toBe(0); + expect(await forgettings()).toEqual([]); + expect(entries.filter(({ level }) => level === "error")).toEqual([]); + }); + + it("holds a kept key it could not destroy, and names when the hold ends", async () => { + const live = await memory().createStorage(); + const { sweep, clock, entries } = await setUp( + { + ...live, + transact: () => Promise.reject(new Error("store down")), + }, + { retentionMs: null, module: { categories: {} } }, + ); + await live.personalKeys.add("c-1", "orders", testPersonalKey("1")); + await live.personalKeys.retain("c-1", testPersonalKey("1").fingerprint, { + since: day(0), + lastActiveAt: day(0), + }); + expect(await sweep.nextDueAt()).toEqual(new Date(day(0))); + expect(await sweep.sweep(clock.now())).toBe(0); + expect(entries).toContainEqual( + expect.objectContaining({ + level: "error", + message: "personal data past the retention could not be forgotten", + fields: expect.objectContaining({ personId: "c-1", category: "orders" }), + }), + ); + const [held] = await live.personalKeys.listRetained({ except: [], limit: 1 }); + expect(held?.hold).not.toBeNull(); + expect(await sweep.nextDueAt()).toEqual(new Date(held?.hold?.until ?? "")); + }); + it("logs a person it cannot forget and goes on with the next", async () => { const live = await memory().createStorage(); let broken = true; @@ -490,7 +692,7 @@ describe("createRetentionSweep", () => { expect(await forgottenFor()).toEqual([["c-2", "retention"]]); expect(entries).toContainEqual({ level: "error", - message: "person past the retention could not be forgotten", + message: "personal data past the retention could not be forgotten", fields: expect.objectContaining({ personId: "c-1", errorMessage: "disk full" }), }); await record(); @@ -560,7 +762,7 @@ describe("createRetentionSweep", () => { expect(await sweep.sweep(clock.now())).toBe(0); expect(entries).toContainEqual({ level: "error", - message: "persons past the retention could not be listed", + message: "personal data past the retention could not be listed", fields: expect.objectContaining({ errorMessage: "connection reset" }), }); expect(await sweep.sweep(clock.now())).toBe(0); @@ -666,7 +868,7 @@ describe("createRetentionSweep", () => { expect(await sweep.sweep(clock.now())).toBe(0); expect(entries).toContainEqual({ level: "error", - message: "person past the retention could not be held", + message: "personal data past the retention could not be held", fields: expect.objectContaining({ personId: "c-3", errorMessage: "read-only" }), }); expect((await storage.persons.get("c-3"))?.hold).toBeNull(); @@ -707,7 +909,7 @@ describe("createRetentionSweep", () => { }, }, intentions: withoutLedger({ ids: toldIds, clock }), - retentionMs: 30 * DAY, + categories: categoriesOf(30 * DAY), ids: toldIds, clock, logger: createRecordingLogger().logger, @@ -743,7 +945,7 @@ describe("createRetentionSweep", () => { const longer = createRetentionSweep({ storage, intentions: withoutLedger({ ids: longerIds, clock }), - retentionMs: 60 * DAY, + categories: categoriesOf(60 * DAY), ids: longerIds, clock, logger: createRecordingLogger().logger, @@ -777,6 +979,199 @@ describe("createRetentionSweep", () => { expect(await sweep.sweep(clock.now())).toBe(0); expect(asked).toEqual([]); }); + + it("destroys only the key of a declared category idle past its retention, with no retention of the app", async () => { + const { storage, sweep, record, touch, forgettings, advance, clock, entries } = await setUp( + undefined, + { + retentionMs: null, + module: { categories: { profile: { basis: "consent", retention: "30d" } } }, + }, + ); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "profile", testPersonalKey("2")); + await touch("c-1"); + await record(); + expect(await sweep.nextDueAt()).toEqual(new Date(Date.parse(day(30)))); + advance(31); + expect(await sweep.sweep(clock.now())).toBe(1); + expect((await storage.personalKeys.list("c-1")).map(({ category }) => category)).toEqual([ + "default", + ]); + const [event] = await forgettings(); + expect(event?.payload).toEqual({ + personId: "c-1", + reason: "retention", + whole: false, + categories: ["profile"], + retained: [], + expired: [], + fingerprints: ["f-2"], + cancelledCommands: [], + removedDeadLetters: [], + }); + expect(entries).toContainEqual({ + level: "info", + message: "personal data forgotten", + fields: expect.objectContaining({ personId: "c-1", whole: false, destroyedKeys: 1 }), + }); + await record(); + expect((await storage.persons.get("c-1"))?.forgottenAt).toBeNull(); + expect(await sweep.sweep(clock.now())).toBe(0); + expect(await sweep.nextDueAt()).toBeNull(); + }); + + it("counts a retention in calendar months to the very moment it ends, never a day early", async () => { + const { storage, sweep, record, touch, clock } = await setUp(undefined, { + retentionMs: null, + module: { categories: { profile: { basis: "consent", retention: "1mo" } } }, + }); + await storage.personalKeys.add("c-1", "profile", testPersonalKey("2")); + await touch("c-1"); + await record(); + const lastActiveAt = (await storage.personalKeys.find("c-1", testPersonalKey("2").fingerprint)) + ?.lastActiveAt; + const ends = retentionEnd(Date.parse(lastActiveAt ?? ""), { months: 1, ms: 0 }); + expect(await sweep.nextDueAt()).toEqual(new Date(ends)); + clock.set(new Date(ends - 1)); + expect(await sweep.sweep(clock.now())).toBe(0); + clock.set(new Date(ends)); + expect(await sweep.sweep(clock.now())).toBe(1); + }); + + it("destroys a key retained under a legal obligation once its retention runs out from the activity it froze", async () => { + const { storage, sweep, record, touch, forgettings, advance, clock, ids, logger, categories } = + await setUp(undefined, { + retentionMs: null, + module: { categories: { invoices: { basis: "legal-obligation", retention: "30d" } } }, + }); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("1")); + await touch("c-1"); + await record(); + advance(10); + await forgetPersonalData({ + storage, + intentions: withoutLedger({ ids, clock }), + ids, + clock, + logger, + concurrencyRetries: 0, + categories, + batchSize: 2, + personId: "c-1", + }); + await touch("c-1"); + await record(); + expect(await storage.personalKeys.list("c-1")).toEqual([ + { + category: "invoices", + key: testPersonalKey("1"), + retainedSince: day(10), + lastActiveAt: day(0), + }, + ]); + expect(await sweep.nextDueAt()).toEqual(new Date(Date.parse(day(30)))); + advance(19); + expect(await sweep.sweep(clock.now())).toBe(0); + advance(1); + expect(await sweep.sweep(clock.now())).toBe(1); + expect(await storage.personalKeys.list("c-1")).toEqual([]); + expect((await forgettings()).map(({ payload }) => payload)).toEqual([ + expect.objectContaining({ reason: "requested", whole: true, retained: ["invoices"] }), + expect.objectContaining({ + reason: "retention", + whole: false, + categories: [], + retained: [], + expired: ["invoices"], + }), + ]); + }); + + it("destroys at once what a forgetting kept once no legal obligation keeps its category", async () => { + const live = await memory().createStorage(); + const legal = { + categories: { + invoices: { basis: "legal-obligation", retention: "6y" }, + orders: { basis: "legal-obligation", retention: "6y" }, + }, + } as const; + const kept = await setUp(live, { retentionMs: null, module: legal }); + await live.personalKeys.add("c-1", "invoices", testPersonalKey("1")); + await live.personalKeys.add("c-2", "orders", testPersonalKey("2")); + await kept.touch("c-1"); + await kept.touch("c-2"); + await kept.record(); + for (const person of ["c-1", "c-2"]) { + await forgetPersonalData({ + storage: live, + intentions: withoutLedger({ ids: kept.ids, clock: kept.clock }), + ids: kept.ids, + clock: kept.clock, + logger: kept.logger, + concurrencyRetries: 0, + categories: buildPersonalCategories({ module: legal, retention: null }), + batchSize: 2, + personId: person, + }); + } + expect(await kept.sweep.sweep(kept.clock.now())).toBe(0); + expect(await kept.sweep.nextDueAt()).not.toEqual(kept.clock.now()); + + const changed = await setUp(live, { + retentionMs: null, + module: { categories: { invoices: { basis: "contract", retention: "6y" } } }, + }); + expect(await changed.sweep.nextDueAt()).toEqual(new Date(day(0))); + expect(await changed.sweep.sweep(changed.clock.now())).toBe(2); + expect(await live.personalKeys.list("c-1")).toEqual([]); + expect(await live.personalKeys.list("c-2")).toEqual([]); + expect((await changed.forgettings()).map(({ payload }) => payload).slice(-2)).toEqual([ + expect.objectContaining({ reason: "retention", whole: false, expired: ["invoices"] }), + expect.objectContaining({ reason: "retention", whole: false, expired: ["orders"] }), + ]); + expect(await changed.sweep.nextDueAt()).toBeNull(); + }); + + it("holds a key it could not destroy on the key, so the person's other keys and row stay free", async () => { + const live = await memory().createStorage(); + let failing = true; + const { storage, sweep, record, touch, advance, clock } = await setUp( + failingForgets(live, () => failing), + { module: { categories: { profile: { basis: "consent", retention: "10d" } } } }, + ); + await storage.personalKeys.add("c-1", "profile", testPersonalKey("1")); + await touch("c-1"); + await record(); + advance(11); + const failedAt = clock.now().getTime(); + expect(await sweep.sweep(clock.now())).toBe(0); + expect(await storage.personalKeys.listDue({ category: "profile", limit: 10 })).toEqual([ + expect.objectContaining({ + fingerprint: "f-1", + hold: { until: new Date(failedAt + HOLD_MS).toISOString(), attempts: 1 }, + }), + ]); + expect((await storage.persons.get("c-1"))?.hold).toBeNull(); + expect(await sweep.nextDueAt()).toEqual(new Date(failedAt + HOLD_MS)); + failing = false; + clock.set(new Date(failedAt + HOLD_MS)); + expect(await sweep.sweep(clock.now())).toBe(1); + expect(await storage.personalKeys.list("c-1")).toEqual([]); + expect(await sweep.nextDueAt()).toEqual(new Date(Date.parse(day(30)))); + }); +}); + +describe("retains", () => { + it("says whether the app's retention or any declared category forgets for inactivity", () => { + expect(retains(categoriesOf(null))).toBe(false); + expect(retains(categoriesOf(DAY))).toBe(true); + expect( + retains( + categoriesOf(null, { categories: { profile: { basis: "consent", retention: "1d" } } }), + ), + ).toBe(true); + }); }); describe("the retention and the forget ledger", () => { @@ -811,14 +1206,14 @@ describe("the retention and the forget ledger", () => { const sweep = createRetentionSweep({ storage, intentions, - retentionMs: 30 * DAY, + categories: categoriesOf(30 * DAY), ids, clock, logger, concurrencyRetries: 1, batchSize: 2, }); - await storage.personalKeys.add("c-1", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); await touch("c-1"); await touch("c-2"); await record(); @@ -831,6 +1226,7 @@ describe("the retention and the forget ledger", () => { expect(recorded).toEqual([ { personId: "c-1", + categories: null, eventId: forgetting?.id, forgottenAt: forgetting?.timestamp, reason: "retention", @@ -842,11 +1238,11 @@ describe("the retention and the forget ledger", () => { it("says a person is due as a sweep would forget them now, by their row and what it has not recorded", async () => { const { record, touch, advance, clock, storage } = await setUp(); const due = retentionDueOf({ - retentionMs: 30 * DAY, + categories: categoriesOf(30 * DAY), batchSize: 2, checkpoints: storage.checkpointStore, }); - const isDue = (person: string) => due(storage, person, clock.now()); + const isDue = (person: string) => due(storage, person, clock.now(), null); for (const person of ["c-1", "c-2", "c-3", "c-4"]) await touch(person); await record(); expect(await isDue("c-1")).toBe(false); @@ -861,6 +1257,46 @@ describe("the retention and the forget ledger", () => { expect(await isDue("c-4")).toBe(false); }); + it("says keys are due as their category's retention would forget them now, every one of them", async () => { + const { record, touch, advance, clock, storage } = await setUp(undefined, { + retentionMs: null, + module: { + categories: { + profile: { basis: "consent", retention: "30d" }, + invoices: { basis: "legal-obligation", retention: "60d" }, + }, + }, + }); + const due = retentionDueOf({ + categories: categoriesOf(null, { + categories: { + profile: { basis: "consent", retention: "30d" }, + invoices: { basis: "legal-obligation", retention: "60d" }, + }, + }), + batchSize: 2, + checkpoints: storage.checkpointStore, + }); + await storage.personalKeys.add("c-1", "profile", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("2")); + await storage.personalKeys.add("c-1", "retired", testPersonalKey("3")); + await touch("c-1"); + await record(); + advance(30); + const keys = async (...categories: string[]) => + (await storage.personalKeys.list("c-1")).filter(({ category }) => + categories.includes(category), + ); + const isDue = async (...categories: string[]) => + due(storage, "c-1", clock.now(), await keys(...categories)); + expect(await isDue("profile")).toBe(true); + expect(await isDue("profile", "invoices")).toBe(false); + expect(await isDue("retired")).toBe(false); + expect(await due(storage, "c-1", clock.now(), null)).toBe(false); + await storage.persons.restrict("c-1", new Date(day(30))); + expect(await isDue("profile")).toBe(false); + }); + // A sweep through a store that runs `meanwhile` once, between the list and the person's forget, // or before the forget's transaction. const racing = async ( @@ -894,14 +1330,14 @@ describe("the retention and the forget ledger", () => { const sweep = createRetentionSweep({ storage: raced, intentions, - retentionMs: 30 * DAY, + categories: categoriesOf(30 * DAY), ids, clock, logger, concurrencyRetries: 1, batchSize: 2, }); - await storage.personalKeys.add("c-1", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "default", testPersonalKey("1")); await setup.touch("c-1"); await setup.record(); setup.advance(31); @@ -926,13 +1362,13 @@ describe("the retention and the forget ledger", () => { expect(await restricted.forgottenFor()).toEqual([]); const rekeyed = await racing("transaction", async (storage) => { - await storage.personalKeys.remove("c-1"); - await storage.personalKeys.add("c-1", testPersonalKey("2")); + await storage.personalKeys.remove("c-1", "f-1"); + await storage.personalKeys.add("c-1", "default", testPersonalKey("2")); }); expect(await rekeyed.sweep.sweep(rekeyed.clock.now())).toBe(0); expect(rekeyed.entries.filter(({ level }) => level === "error")).toEqual([]); expect(await rekeyed.forgottenFor()).toEqual([]); - expect(await rekeyed.storage.personalKeys.get("c-1")).toEqual(testPersonalKey("2")); + expect(await rekeyed.storage.personalKeys.get("c-1", "default")).toEqual(testPersonalKey("2")); const [row] = await rekeyed.storage.persons.listIdle({ limit: 1 }); expect(row?.hold).not.toBeNull(); expect(await rekeyed.sweep.sweep(rekeyed.clock.now())).toBe(0); @@ -962,18 +1398,18 @@ describe("the retention and the forget ledger", () => { expect(await sweep.sweep(clock.now())).toBe(0); expect((await forgottenFor()).map(([person]) => person)).toEqual(["c-1"]); expect(intended).not.toBe(""); - expect(await storage.personalKeys.get("c-1")).not.toBeNull(); + expect(await storage.personalKeys.get("c-1", "default")).not.toBeNull(); }); it("says a person whose row never saw them active is not due", async () => { const { storage, clock } = await setUp(); await storage.persons.record({ person: "c-5", position: 1, forgottenAt: null }); const due = retentionDueOf({ - retentionMs: DAY, + categories: categoriesOf(DAY), batchSize: 2, checkpoints: storage.checkpointStore, }); - expect(await due(storage, "c-5", new Date(clock.now().getTime() + 2 * DAY))).toBe(false); + expect(await due(storage, "c-5", new Date(clock.now().getTime() + 2 * DAY), null)).toBe(false); }); it("records no intention for a person active since their row was recorded", async () => { @@ -1070,6 +1506,43 @@ describe("the retention across instances", () => { expect((await forgottenFor()).map(([person]) => person)).toEqual(["c-1", "c-2", "c-3", "c-4"]); }); + it("leaves the categories after the one it lost its lease in to the instance that took it", async () => { + const set = await setUp(undefined, { + retentionMs: null, + module: { categories: { profile: { basis: "consent", retention: "10d" } } }, + }); + const { storage, sweepOver, clock } = set; + for (const person of ["c-1", "c-2"]) { + await storage.personalKeys.add(person, "profile", testPersonalKey(person)); + await set.touch(person); + } + await storage.personalKeys.add("c-9", "retired", testPersonalKey("9")); + await storage.personalKeys.retain("c-9", "f-9", { since: day(0), lastActiveAt: day(0) }); + await set.record(); + set.advance(11); + const other = sweepOver(storage); + let retainedLists = 0; + let handedOver = 0; + const stalling = committing(storage, async () => { + clock.set(new Date(clock.now().getTime() + LEASE_MS)); + handedOver = await other.sweep(clock.now()); + }); + const one = sweepOver({ + ...stalling, + personalKeys: { + ...stalling.personalKeys, + listRetained: (args) => { + retainedLists += 1; + return stalling.personalKeys.listRetained(args); + }, + }, + }); + expect(await one.sweep(clock.now())).toBe(1); + expect(handedOver).toBe(2); + expect(retainedLists).toBe(0); + expect(await storage.personalKeys.list("c-9")).toEqual([]); + }); + it("clears the holds a restore brought back, on the instance that found it first", async () => { const { storage, sweepOver, clock } = await idle(["c-1"]); const hold = { @@ -1116,25 +1589,53 @@ describe("the retention across instances", () => { }).restored(true); expect(entries).toContainEqual({ level: "error", - message: "the holds of persons past the retention could not be cleared", + message: "the holds of personal data past the retention could not be cleared", fields: expect.objectContaining({ errorMessage: "cannot write" }), }); }); - it("says how many persons are past the retention and when the next one comes due", async () => { - const { storage, sweepOver, touch, record, advance, clock } = await setUp(); + it("says, for each retention, how many are past it and when the next one comes due", async () => { + const module: PersonalDataModule = { + categories: { + profile: { basis: "consent", retention: "10d" }, + invoices: { basis: "legal-obligation", retention: "6y" }, + }, + }; + const { storage, sweepOver, touch, record, advance, clock } = await setUp(undefined, { + module, + }); const sweep = sweepOver(storage); - expect(await sweep.status()).toEqual({ retentionMs: 30 * DAY, due: 0, nextDueAt: null }); + const status = ( + app: readonly [number, string | null], + profile: readonly [number, string | null], + invoices: readonly [number, string | null], + keptBeyond = 0, + ) => ({ + categories: [ + { category: "default", retention: "30d", due: app[0], nextDueAt: app[1] }, + { category: "profile", retention: "10d", due: profile[0], nextDueAt: profile[1] }, + { category: "invoices", retention: "6y", due: invoices[0], nextDueAt: invoices[1] }, + ], + keptBeyond, + }); + expect(await sweep.status()).toEqual(status([0, null], [0, null], [0, null])); + await storage.personalKeys.add("c-1", "profile", testPersonalKey("1")); + await storage.personalKeys.add("c-1", "invoices", testPersonalKey("2")); + await storage.personalKeys.add("c-3", "retired", testPersonalKey("3")); + await storage.personalKeys.retain("c-3", "f-3", { since: day(0), lastActiveAt: day(0) }); await touch("c-1"); advance(1); await touch("c-2"); await record(); advance(29.5); - expect(await sweep.status()).toEqual({ - retentionMs: 30 * DAY, - due: 1, - nextDueAt: new Date(Date.parse(day(0)) + 30 * DAY).toISOString(), - }); + expect(await sweep.status()).toEqual( + status( + [1, new Date(Date.parse(day(0)) + 30 * DAY).toISOString()], + [1, new Date(Date.parse(day(0)) + 10 * DAY).toISOString()], + [0, "2032-01-01T00:00:00.000Z"], + 1, + ), + ); expect(clock.now().getTime()).toBe(Date.parse(day(30)) + DAY / 2); }); }); diff --git a/packages/core/src/kernel/personal-data/retention.ts b/packages/core/src/kernel/personal-data/retention.ts index f3e5ca3c..053245e0 100644 --- a/packages/core/src/kernel/personal-data/retention.ts +++ b/packages/core/src/kernel/personal-data/retention.ts @@ -1,20 +1,32 @@ -import type { Storage } from "../../adapter/adapter.ts"; +import type { Storage, StorageTransaction } from "../../adapter/adapter.ts"; import type { CheckpointStore } from "../../adapter/storage/checkpoint-store.ts"; import type { EventStore } from "../../adapter/storage/event-store.ts"; +import type { Hold } from "../../adapter/storage/hold.ts"; import { isReleasedBy } from "../../adapter/storage/hold.ts"; import type { PersonActivity, PersonStore, StoredPerson, } from "../../adapter/storage/person-store.ts"; +import { + DEFAULT_PERSONAL_CATEGORY, + type DueKey, + type PersonalKeyStore, +} from "../../adapter/storage/personal-key-store.ts"; import type { Clock } from "../../contracts/clock.ts"; +import { type Retention, retentionEnd, retentionStart } from "../../contracts/duration.ts"; import type { StoredEvent } from "../../contracts/event.ts"; import type { IdGenerator } from "../../contracts/ids.ts"; import type { Logger } from "../../contracts/logger.ts"; -import { BOUNDA_AGGREGATE, PERSONAL_DATA_FORGOTTEN_EVENT } from "../../modules/bounda-events.ts"; +import { + PERSONAL_DATA_FORGOTTEN_EVENT, + type PersonalDataForgottenPayload, +} from "../../modules/bounda-events.ts"; import type { Subscriber } from "../dispatch/delivery.ts"; import { createLease } from "../shared/lease.ts"; -import { forgetWithin, logForgotten, STALE_INTENTION } from "./forget.ts"; +import { isActivity } from "./activity.ts"; +import { formatRetention, type PersonalCategories, type PersonalCategory } from "./categories.ts"; +import { type ForgetScope, forgetWithin, logForgotten, STALE_INTENTION } from "./forget.ts"; import { createPause, holdAfter } from "./hold.ts"; import type { ForgetIntentions } from "./intention.ts"; import type { RetentionDueFunction } from "./reconciliation.ts"; @@ -22,7 +34,7 @@ import { transactRetrying } from "./transact.ts"; export const RETENTION_SUBSCRIBER: "retention" = "retention"; -// How many persons a pass of the worker forgets at most. +// How many persons, and how many keys of each category, a pass of the worker forgets at most. export const RETENTION_SWEEP_LIMIT = 50; /** @@ -31,20 +43,37 @@ export const RETENTION_SWEEP_LIMIT = 50; export const RETENTION_LEASE = "personal-data:retention:lease"; /** - * Where the retention stands: how many persons are idle past it now, held ones included, and when + * Where one retention stands: the app's, of persons as a whole, under the default category, or a + * declared category's, of its keys. How many are idle past it now, held ones included, and when * the next one comes due. */ -export interface RetentionStatus { - readonly retentionMs: number; +export interface CategoryRetentionStatus { + readonly category: string; + /** + * As `app/personal-data.ts` writes it, `"30d"` or `"6y"`. + */ + readonly retention: string; readonly due: number; readonly nextDueAt: string | null; } -const isActivity = (event: StoredEvent): boolean => event.aggregateType !== BOUNDA_AGGREGATE; +/** + * Where the retention stands: each retention set, and how many keys a forgetting kept that no + * legal obligation keeps any more, which the sweep destroys at once. + */ +export interface RetentionStatus { + readonly categories: readonly CategoryRetentionStatus[]; + readonly keptBeyond: number; +} // Asked only of the runtime's own events, once `isActivity` said no. const isForget = (event: StoredEvent): boolean => event.type === PERSONAL_DATA_FORGOTTEN_EVENT; +// A forgetting of the person as a whole, as every one recorded before categories was; one of some +// categories leaves the row as it was. +const isWholeForget = (event: StoredEvent): boolean => + isForget(event) && (event.payload as PersonalDataForgottenPayload).whole !== false; + // What a `PersonalDataForgotten` says of its person; their last activity stays as it was. const forgettingOf = (event: StoredEvent & { readonly person: string }): PersonActivity => ({ person: event.person, @@ -75,10 +104,10 @@ const isIdle = (row: StoredPerson | null): row is StoredPerson => row !== null && row.restrictedSince === null && row.forgottenAt === null; export interface RetentionDueArgs { - readonly retentionMs: number; + readonly categories: PersonalCategories; readonly batchSize: number; /** - * Where the retention subscriber keeps how far its rows reach. + * Where the retention subscriber keeps how far its records reach. */ readonly checkpoints: CheckpointStore; } @@ -87,22 +116,36 @@ export interface RetentionDueOfFunction { (args: RetentionDueArgs): RetentionDueFunction; } +const isPast = (lastActiveAt: string | null, retention: Retention | null, now: Date): boolean => + lastActiveAt !== null && + retention !== null && + retentionEnd(Date.parse(lastActiveAt), retention) <= now.getTime(); + /** - * Whether the retention would forget the person at `now`, as a sweep decides it in the transaction - * that forgets them. + * Whether the retention would forget at `now` the person as a whole, past the app's retention, or + * each of the keys given, past its category's, as a sweep decides it in the transaction that + * forgets them. */ export const retentionDueOf: RetentionDueOfFunction = - ({ retentionMs, batchSize, checkpoints }) => - async (stores, personId, now) => { + ({ categories, batchSize, checkpoints }) => + async (stores, personId, now, keys) => { const row = await stores.persons.get(personId); - if (!isIdle(row) || row.lastActiveAt === null) return false; - if (Date.parse(row.lastActiveAt) + retentionMs > now.getTime()) return false; + const past = + keys === null + ? isIdle(row) && + isPast(row.lastActiveAt, categories.of(DEFAULT_PERSONAL_CATEGORY)?.retention ?? null, now) + : (row?.restrictedSince ?? null) === null && + keys.every(({ category, lastActiveAt }) => + isPast(lastActiveAt, categories.of(category)?.retention ?? null, now), + ); + if (!past) return false; const through = await checkpoints.get(RETENTION_SUBSCRIBER); return rowIsCurrent(stores.eventStore, personId, through, batchSize); }; export interface CreateRetentionSubscriberArgs { readonly persons: PersonStore; + readonly keys: Pick; } export interface CreateRetentionSubscriberFunction { @@ -111,12 +154,16 @@ export interface CreateRetentionSubscriberFunction { /** * Follows the global stream from its first event and records, per person an event names, their - * last activity, an event of the app's aggregates, and their last forgetting, which activity - * after it clears; the runtime's other events on `bounda:` change nothing. One write per - * person and batch, which the store ignores for a batch it recorded already, so a batch delivered - * twice, or from the start again, leaves the same rows. + * last activity, an event of the app's aggregates, on their row and on each of their keys in + * force, and their last forgetting as a whole, which activity after it clears; the runtime's + * other events on `bounda:` change nothing. One write per person and batch, which the + * stores ignore for a batch they recorded already, so a batch delivered twice, or from the start + * again, leaves the same rows. */ -export const createRetentionSubscriber: CreateRetentionSubscriberFunction = ({ persons }) => ({ +export const createRetentionSubscriber: CreateRetentionSubscriberFunction = ({ + persons, + keys, +}) => ({ name: RETENTION_SUBSCRIBER, kind: "retention", process: async (events) => { @@ -132,26 +179,32 @@ export const createRetentionSubscriber: CreateRetentionSubscriberFunction = ({ p lastActiveAt: event.timestamp, forgottenAt: null, }); - } else if (isForget(event)) { + } else if (isWholeForget(event)) { recorded.set(person, { ...known, ...forgettingOf({ ...event, person }) }); } } - for (const activity of recorded.values()) await persons.record(activity); + for (const activity of recorded.values()) { + await persons.record(activity); + if (activity.lastActiveAt !== undefined) { + await keys.touch(activity.person, activity.lastActiveAt); + } + } return events.length; }, }); export interface RetentionSweep { /** - * Forgets, as `forget` does with `reason: "retention"`, up to `RETENTION_SWEEP_LIMIT` persons - * idle past the retention at `now` and not held, and resolves to how many it forgot. Never - * rejects: a person it could not forget is logged and held on their row, longer after each - * failure. + * Forgets, with `reason: "retention"`, up to `RETENTION_SWEEP_LIMIT` persons idle past the app's + * retention at `now` and not held, as a whole, as many keys of each declared category idle past + * that category's retention, and as many keys a forgetting kept that no legal obligation keeps + * any more, and resolves to how many forgettings it made. Never rejects: + * a person or a key it could not forget is logged and held, longer after each failure. */ sweep(now: Date): Promise; /** - * When the next person the retention may forget becomes due, a held one not before their hold - * ends; `null` when there is none. + * When the next person or key the retention may forget becomes due, a held one not before its + * hold ends; `null` when there is none. */ nextDueAt(): Promise; status(): Promise; @@ -169,14 +222,14 @@ export interface RetentionSweep { export interface CreateRetentionSweepArgs { readonly storage: Storage; readonly intentions: ForgetIntentions; - readonly retentionMs: number; + readonly categories: PersonalCategories; readonly ids: IdGenerator; readonly clock: Clock; readonly logger: Logger; readonly concurrencyRetries: number; readonly batchSize: number; /** - * Called with each person forgotten, once their forgetting is committed. + * Called with each person a forgetting covered, once it is committed. */ readonly forgotten?: (personId: string) => void; } @@ -185,18 +238,62 @@ export interface CreateRetentionSweepFunction { (args: CreateRetentionSweepArgs): RetentionSweep; } +type Outcome = "forgotten" | "skipped" | "undone"; + +// Where a forgetting is checked: on the store before its intention, and again in its transaction. +type CheckedStores = Pick; + +interface Swept { + readonly forgotten: number; + readonly kept: boolean; + // Another instance took the lease: the pass ends. + readonly lost: boolean; +} + +// What a list of things to forget is read through: persons idle past the app's retention, or the +// keys of one category idle past its own. +interface Sweepable { + // The latest last activity due at `at`; every one is when omitted. + activeBefore(at: number): Date | undefined; + // When one listed comes due; `null` for one with no activity recorded. + deadlineOf(due: Due): number | null; + list(args: { + readonly activeBefore?: Date; + readonly releasedBy?: Date; + readonly limit: number; + }): Promise; + // How many `list` lists with `activeBefore`, whatever their holds. + count(activeBefore: Date | undefined): Promise; + nextHoldEnd(after: Date): Promise; + holdOf(due: Due): Hold | null; + // Resolves `undone` for one whose record is behind, left for a later pass, and `skipped` for + // one restricted, or forgotten by another instance, since the list was read. + forget(due: Due, through: number): Promise; + hold(due: Due, hold: Hold): Promise; + // For the log. + describe(due: Due): Readonly>; +} + +interface Runner { + sweep(now: Date, through: () => Promise): Promise; + nextDue(now: Date): Promise; + // For a retention of the app or a category; the keys kept beyond a legal obligation have none. + readonly status?: (now: Date) => Promise; +} + /** - * The rows lag behind the stream by what the subscriber has not recorded yet, so a person is - * forgotten only when no event past its checkpoint names them as active or forgotten already, - * read in the transaction that forgets them, with their restriction: a person active a moment - * ago is not forgotten for an activity the row does not show yet, and of two instances sweeping - * one person, the one whose append conflicts reads the other's forgetting on its retry and stops. - * `storage` is the store as written: the sweep reads no personal field. + * The records lag behind the stream by what the subscriber has not recorded yet, so a person, or + * a key in force, is forgotten only when no event past its checkpoint names them as active or + * forgotten already, read in the transaction that forgets them, with their restriction: a person + * active a moment ago is not forgotten for an activity the record does not show yet, and of two + * instances sweeping one person, the one whose append conflicts reads the other's forgetting on + * its retry and stops. A key kept past a forgetting follows no activity, so it needs no such + * check. `storage` is the store as written: the sweep reads no personal field. */ export const createRetentionSweep: CreateRetentionSweepFunction = ({ storage, intentions, - retentionMs, + categories, ids, clock, logger, @@ -204,31 +301,32 @@ export const createRetentionSweep: CreateRetentionSweepFunction = ({ batchSize, forgotten: onForgotten, }) => { - // Resolves `undone` for a person whose row is behind, or who was given a key since the - // intention, who is then left for a later pass, and `skipped` for one restricted, or forgotten by - // another instance, since the list was read. Both are decided once before the intention, so a - // person plainly left leaves none, and again in the transaction; an intention it then leaves is - // committed again only if the retention would still forget the person. - const forget = async ( + // Resolves `undone` for one whose record is behind, or who was given a key since the intention, + // left for a later pass, and `skipped` for one restricted, or forgotten by another instance, + // since the list was read. Both are decided once before the intention, so one plainly left + // leaves none, and again in the transaction; an intention it then leaves is committed again + // only if the retention would still forget it. + const forgetting = async ( personId: string, - through: number, - ): Promise<"forgotten" | "skipped" | "undone"> => { - if (!isIdle(await storage.persons.get(personId))) return "skipped"; - if (!(await rowIsCurrent(storage.eventStore, personId, through, batchSize))) return "undone"; + scope: ForgetScope, + check: (stores: CheckedStores) => Promise<"go" | Exclude>, + whole: boolean, + ): Promise => { + const before = await check(storage); + if (before !== "go") return before; const intention = await intentions.intend({ personId, reason: "retention", + scope, keys: storage.personalKeys, }); const correlationId = ids.next(); - const forgotten = await transactRetrying({ + const outcome = await transactRetrying({ storage, concurrencyRetries, work: async (transaction) => { - if (!isIdle(await transaction.persons.get(personId))) return "skipped"; - if (!(await rowIsCurrent(transaction.eventStore, personId, through, batchSize))) { - return "undone"; - } + const checked = await check(transaction); + if (checked !== "go") return checked; const outcome = await forgetWithin({ transaction, ids, @@ -236,23 +334,133 @@ export const createRetentionSweep: CreateRetentionSweepFunction = ({ personId, intention, context: { correlationId, causationId: correlationId, depth: 0 }, + categories, + scope, + batchSize, }); if (outcome === STALE_INTENTION) return "undone"; if ("held" in outcome) return "skipped"; // Recorded here as the subscriber will record it, so the row leaves the idle list, its - // hold with it, before any pass can take the person again; the subscriber's record of the - // same event then changes nothing. - await transaction.persons.record(forgettingOf({ ...outcome.event, person: personId })); + // hold with it, before any pass can take the person again; the subscriber's record of + // the same event then changes nothing. + if (whole) { + await transaction.persons.record(forgettingOf({ ...outcome.event, person: personId })); + } return outcome; }, }); - if (typeof forgotten === "string") return forgotten; - logForgotten(logger, forgotten); + if (typeof outcome === "string") return outcome; + logForgotten(logger, outcome); onForgotten?.(personId); return "forgotten"; }; - // When the store itself fails a pass, so no row can keep the wait, the sweep stops in this + const persons = (retention: Retention): Sweepable => ({ + activeBefore: (at) => new Date(retentionStart(at, retention)), + deadlineOf: ({ lastActiveAt }) => + lastActiveAt === null ? null : retentionEnd(Date.parse(lastActiveAt), retention), + list: (args) => storage.persons.listIdle(args), + count: (activeBefore) => + storage.persons.countIdle(activeBefore === undefined ? {} : { activeBefore }), + nextHoldEnd: (after) => storage.persons.nextHoldEnd(after), + holdOf: (person) => person.hold, + forget: (person, through) => + forgetting( + person.person, + { kind: "whole" }, + async (transaction) => { + const row = await transaction.persons.get(person.person); + if (row === null || row.restrictedSince !== null || row.forgottenAt !== null) { + return "skipped"; + } + return (await rowIsCurrent(transaction.eventStore, person.person, through, batchSize)) + ? "go" + : "undone"; + }, + true, + ), + hold: (person, hold) => storage.persons.hold(person.person, person.lastActiveAt ?? "", hold), + describe: (person) => ({ personId: person.person }), + }); + + const keysOf = (category: PersonalCategory & { readonly retention: Retention }) => + ({ + activeBefore: (at) => new Date(retentionStart(at, category.retention)), + deadlineOf: ({ lastActiveAt }) => retentionEnd(Date.parse(lastActiveAt), category.retention), + list: (args) => storage.personalKeys.listDue({ ...args, category: category.name }), + count: (activeBefore) => + storage.personalKeys.countDue({ + category: category.name, + ...(activeBefore === undefined ? {} : { activeBefore }), + }), + nextHoldEnd: (after) => + storage.personalKeys.nextDueHoldEnd({ category: category.name, after }), + holdOf: (due) => due.hold, + forget: (due, through) => + forgetting( + due.person, + { kind: "keys", fingerprints: [due.fingerprint] }, + async (transaction) => { + const [row, key] = await Promise.all([ + transaction.persons.get(due.person), + transaction.personalKeys.find(due.person, due.fingerprint), + ]); + if ((row?.restrictedSince ?? null) !== null || key?.lastActiveAt !== due.lastActiveAt) { + return "skipped"; + } + if (key.retainedSince !== null) return "go"; + return (await rowIsCurrent(transaction.eventStore, due.person, through, batchSize)) + ? "go" + : "undone"; + }, + false, + ), + hold: (due, hold) => + storage.personalKeys.holdDue(due.person, due.fingerprint, due.lastActiveAt, hold), + describe: (due) => ({ personId: due.person, category: due.category }), + }) satisfies Sweepable; + + // What a forgetting kept that no legal obligation keeps any more, as a category whose basis + // changed or that the app no longer declares, goes at once. + const legal = categories.declared + .filter(({ basis }) => basis === "legal-obligation") + .map(({ name }) => name); + const keptBeyond: Sweepable = { + activeBefore: () => undefined, + deadlineOf: ({ retainedSince }) => Date.parse(retainedSince ?? ""), + list: ({ releasedBy, limit }) => + storage.personalKeys.listRetained({ + except: legal, + ...(releasedBy === undefined ? {} : { releasedBy }), + limit, + }), + count: () => storage.personalKeys.countRetained({ except: legal }), + nextHoldEnd: (after) => storage.personalKeys.nextRetainedHoldEnd({ except: legal, after }), + holdOf: (due) => due.hold, + forget: (due) => + forgetting( + due.person, + { kind: "keys", fingerprints: [due.fingerprint] }, + async (transaction) => { + const [row, key] = await Promise.all([ + transaction.persons.get(due.person), + transaction.personalKeys.find(due.person, due.fingerprint), + ]); + return (row?.restrictedSince ?? null) !== null || + key === null || + key.retainedSince === null || + legal.includes(key.category) + ? "skipped" + : "go"; + }, + false, + ), + hold: (due, hold) => + storage.personalKeys.holdDue(due.person, due.fingerprint, due.lastActiveAt, hold), + describe: (due) => ({ personId: due.person, category: due.category }), + }; + + // When the store itself fails a pass, so no record can keep the wait, the sweep stops in this // instance for a while. const pause = createPause(); const lease = createLease({ @@ -262,76 +470,146 @@ export const createRetentionSweep: CreateRetentionSweepFunction = ({ logger, }); - // A person left undone waits on their own row, so one whose forget keeps failing never holds - // the others back, every instance skips them, and a host that wakes for `nextDueAt` never wakes - // for them again at once. - const hold = async (person: StoredPerson, at: number, failed: boolean): Promise => { + // One left undone waits on its own record, so one whose forget keeps failing never holds the + // others back, every instance skips it, and a host that wakes for `nextDueAt` never wakes for it + // again at once. + const hold = async ( + sweepable: Sweepable, + due: Due, + at: number, + failed: boolean, + ): Promise => { try { - await storage.persons.hold( - person.person, - person.lastActiveAt ?? "", - holdAfter({ held: person.hold, at, failed }), - ); + await sweepable.hold(due, holdAfter({ held: sweepable.holdOf(due), at, failed })); return true; } catch (error) { - logger.error("person past the retention could not be held", { - personId: person.person, + logger.error("personal data past the retention could not be held", { + ...sweepable.describe(due), error, }); return false; } }; - // Resolves whether the store kept up: a person it could not hold pauses the sweep. - const attempt = async ( - person: StoredPerson, + // Resolves whether the store kept up: one it could not hold pauses the sweep. + const attempt = async ( + sweepable: Sweepable, + due: Due, at: number, through: number, ): Promise<{ readonly forgotten: boolean; readonly kept: boolean }> => { try { - const outcome = await forget(person.person, through); + const outcome = await sweepable.forget(due, through); if (outcome !== "undone") return { forgotten: outcome === "forgotten", kept: true }; - return { forgotten: false, kept: await hold(person, at, false) }; + return { forgotten: false, kept: await hold(sweepable, due, at, false) }; } catch (error) { - logger.error("person past the retention could not be forgotten", { - personId: person.person, + logger.error("personal data past the retention could not be forgotten", { + ...sweepable.describe(due), error, }); - return { forgotten: false, kept: await hold(person, at, true) }; + return { forgotten: false, kept: await hold(sweepable, due, at, true) }; } }; - const sweep = async (now: Date): Promise => { + const sweepOne = async ( + sweepable: Sweepable, + now: Date, + through: () => Promise, + ): Promise => { const at = now.getTime(); - // While another instance holds the lease, its row is one read, the persons a batch. - if (at < lease.heldElsewhereUntil() && !(await lease.take(at))) return 0; - const due = await storage.persons.listIdle({ - activeBefore: new Date(at - retentionMs), + const activeBefore = sweepable.activeBefore(at); + const due = await sweepable.list({ + ...(activeBefore === undefined ? {} : { activeBefore }), releasedBy: now, limit: RETENTION_SWEEP_LIMIT, }); - if (due.length === 0) { - pause.resume(); - return 0; - } - const through = await storage.checkpointStore.get(RETENTION_SUBSCRIBER); let forgotten = 0; let kept = true; - for (const person of due) { + for (const one of due) { // A lease that lapsed while this instance stalled was taken by another, which goes on. - if (!(await lease.take(clock.now().getTime()))) break; - const outcome = await attempt(person, at, through); + if (!(await lease.take(clock.now().getTime()))) return { forgotten, kept, lost: true }; + const outcome = await attempt(sweepable, one, at, await through()); if (outcome.forgotten) forgotten += 1; kept &&= outcome.kept; } - pause.settle(at, kept); + return { forgotten, kept, lost: false }; + }; + + const runnerOf = ( + sweepable: Sweepable, + category?: PersonalCategory & { readonly retention: Retention }, + ): Runner => ({ + sweep: (now, through) => sweepOne(sweepable, now, through), + nextDue: (now) => nextDueOf(sweepable, now), + ...(category === undefined + ? {} + : { + status: async (now: Date) => { + const [due, [first]] = await Promise.all([ + sweepable.count(sweepable.activeBefore(now.getTime())), + sweepable.list({ limit: 1 }), + ]); + const deadline = first === undefined ? null : sweepable.deadlineOf(first); + return { + category: category.name, + retention: formatRetention(category.retention), + due, + nextDueAt: deadline === null ? null : new Date(deadline).toISOString(), + }; + }, + }), + }); + + // The default category's retention is the app's, which forgets persons as a whole. + const runners: readonly Runner[] = [ + ...categories.all.flatMap((category) => { + const { retention } = category; + if (retention === null) return []; + return category.basis === null + ? [runnerOf(persons(retention), { ...category, retention })] + : [runnerOf(keysOf({ ...category, retention }), { ...category, retention })]; + }), + runnerOf(keptBeyond), + ]; + + const sweep = async (now: Date): Promise => { + const at = now.getTime(); + // While another instance holds the lease, its row is one read, the records a batch. + if (at < lease.heldElsewhereUntil() && !(await lease.take(at))) return 0; + let checkpoint: Promise | undefined; + const through = (): Promise => { + checkpoint ??= storage.checkpointStore.get(RETENTION_SUBSCRIBER); + return checkpoint; + }; + let forgotten = 0; + let kept = true; + for (const runner of runners) { + const swept = await runner.sweep(now, through); + forgotten += swept.forgotten; + kept &&= swept.kept; + if (swept.lost) break; + } + pause.settle(now.getTime(), kept); return forgotten; }; - const deadlineOf = (person: StoredPerson | undefined): number => - typeof person?.lastActiveAt === "string" - ? Date.parse(person.lastActiveAt) + retentionMs - : Number.POSITIVE_INFINITY; + // Nothing comes due before the least recently active one's deadline, so while that one is not + // held it alone answers. A held one comes due when both its deadline and its hold have passed; + // the hold's end is named alone, so in the rare case of a retention lengthened since, a host + // wakes once too early, finds the hold over, and from then on reads the deadline of those free. + const nextDueOf = async (sweepable: Sweepable, now: Date): Promise => { + const deadlineOf = (due: Due | undefined): number => + (due === undefined ? null : sweepable.deadlineOf(due)) ?? Number.POSITIVE_INFINITY; + const [first] = await sweepable.list({ limit: 1 }); + if (first === undefined || isReleasedBy(sweepable.holdOf(first), now)) { + return deadlineOf(first); + } + const [[free], heldUntil] = await Promise.all([ + sweepable.list({ releasedBy: now, limit: 1 }), + sweepable.nextHoldEnd(now), + ]); + return Math.min(deadlineOf(free), heldUntil?.getTime() ?? Number.POSITIVE_INFINITY); + }; return { sweep: async (now) => { @@ -341,51 +619,49 @@ export const createRetentionSweep: CreateRetentionSweepFunction = ({ return await sweep(now); } catch (error) { pause.pause(at); - logger.error("persons past the retention could not be listed", { error }); + logger.error("personal data past the retention could not be listed", { error }); return 0; } }, - // Nobody comes due before the least recently active person's deadline, so while that person - // is not held it alone answers. A held person comes due when both their deadline and their - // hold have passed; the hold's end is named alone, so in the rare case of a retention - // lengthened since, a host wakes once too early, finds their hold over, and from then on reads - // their deadline with the persons free. nextDueAt: async () => { const now = clock.now(); - const [first] = await storage.persons.listIdle({ limit: 1 }); - let next = deadlineOf(first); - if (first !== undefined && !isReleasedBy(first.hold, now)) { - const [[free], heldUntil] = await Promise.all([ - storage.persons.listIdle({ releasedBy: now, limit: 1 }), - storage.persons.nextHoldEnd(now), - ]); - next = Math.min(deadlineOf(free), heldUntil?.getTime() ?? Number.POSITIVE_INFINITY); - } + const next = Math.min(...(await Promise.all(runners.map((runner) => runner.nextDue(now))))); return next === Number.POSITIVE_INFINITY ? null : new Date(pause.notBefore(Math.max(next, lease.heldElsewhereUntil()))); }, status: async () => { const now = clock.now(); - const [due, next] = await Promise.all([ - storage.persons.countIdle({ activeBefore: new Date(now.getTime() - retentionMs) }), - storage.persons.listIdle({ limit: 1 }), + const [statuses, beyond] = await Promise.all([ + Promise.all(runners.flatMap(({ status }) => (status === undefined ? [] : [status(now)]))), + keptBeyond.count(undefined), ]); - const deadline = deadlineOf(next[0]); - return { - retentionMs, - due, - nextDueAt: deadline === Number.POSITIVE_INFINITY ? null : new Date(deadline).toISOString(), - }; + return { categories: statuses, keptBeyond: beyond }; }, release: lease.release, restored: async (first) => { pause.resume(); lease.reset(); if (!first) return; - await storage.persons.releaseHolds().catch((error: unknown) => { - logger.error("the holds of persons past the retention could not be cleared", { error }); + await Promise.all([ + storage.persons.releaseHolds(), + storage.personalKeys.releaseDueHolds(), + ]).catch((error: unknown) => { + logger.error("the holds of personal data past the retention could not be cleared", { + error, + }); }); }, }; }; + +export interface RetainsFunction { + (categories: PersonalCategories): boolean; +} + +/** + * Whether anything is forgotten for inactivity: the app's retention is set, or a category is + * declared, since every declared category has a retention. + */ +export const retains: RetainsFunction = (categories) => + categories.all.some(({ retention }) => retention !== null); diff --git a/packages/core/src/kernel/personal-data/rotation.test.ts b/packages/core/src/kernel/personal-data/rotation.test.ts index c7112287..411f995b 100644 --- a/packages/core/src/kernel/personal-data/rotation.test.ts +++ b/packages/core/src/kernel/personal-data/rotation.test.ts @@ -1,10 +1,14 @@ import { describe, expect, it } from "vitest"; import type { CheckpointStore } from "../../adapter/storage/checkpoint-store.ts"; -import type { PersonalKeyStore } from "../../adapter/storage/personal-key-store.ts"; +import { + DEFAULT_PERSONAL_CATEGORY, + type PersonalKeyStore, +} from "../../adapter/storage/personal-key-store.ts"; import { createFixedClock } from "../../contracts/clock.ts"; import type { Logger } from "../../contracts/logger.ts"; import { memory } from "../../memory/index.ts"; import { LEASE_MS } from "../shared/lease.ts"; +import { removeKeys } from "../test-support.ts"; import { HOLD_MAX_MS, HOLD_MS } from "./hold.ts"; import { type CryptoKeyWrapper, cryptoKeyWrapperOver } from "./key-wrapper.ts"; import { @@ -125,14 +129,18 @@ const countedMarks = (checkpoints: CheckpointStore) => { // The key as the store keeps it. const addUnder = async (store: PersonalKeyStore, kid: string | null, person: string) => { const wrapper = kid === null ? null : fakeWrapper(kid); - await createPersonalKeyWrapping({ wrapper, clock }).over(store).create(person); - const stored = await store.get(person); + await createPersonalKeyWrapping({ wrapper, clock }) + .over(store) + .create(person, DEFAULT_PERSONAL_CATEGORY); + const stored = await store.get(person, DEFAULT_PERSONAL_CATEGORY); if (stored === null) throw new Error(`no key stored for ${person}`); return stored; }; const kidsOf = async (store: PersonalKeyStore, persons: readonly string[]) => - Promise.all(persons.map(async (person) => (await store.get(person))?.kid)); + Promise.all( + persons.map(async (person) => (await store.get(person, DEFAULT_PERSONAL_CATEGORY))?.kid), + ); describe("createKeyRotation", () => { it("rewraps under the current key what a previous key wrapped and what was kept unwrapped", async () => { @@ -149,11 +157,17 @@ describe("createKeyRotation", () => { "unknown", ]); const keys = wrapping.over(store); - expect((await keys.held("c-1"))?.fingerprint).toBe(old.fingerprint); - expect((await keys.held("c-2"))?.fingerprint).toBe(plain.fingerprint); - expect((await keys.held("c-3"))?.fingerprint).toBe(current.fingerprint); - expect((await store.get("c-3"))?.key).toBe(current.key); - expect((await store.get("c-4"))?.fingerprint).toBe(other.fingerprint); + expect((await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY))?.fingerprint).toBe(old.fingerprint); + expect((await keys.held("c-2", DEFAULT_PERSONAL_CATEGORY))?.fingerprint).toBe( + plain.fingerprint, + ); + expect((await keys.held("c-3", DEFAULT_PERSONAL_CATEGORY))?.fingerprint).toBe( + current.fingerprint, + ); + expect((await store.get("c-3", DEFAULT_PERSONAL_CATEGORY))?.key).toBe(current.key); + expect((await store.get("c-4", DEFAULT_PERSONAL_CATEGORY))?.fingerprint).toBe( + other.fingerprint, + ); }); it("rewraps at most ROTATION_BATCH keys a pass", async () => { @@ -204,7 +218,7 @@ describe("createKeyRotation", () => { const gone = await setUp({ ...plain, wrap: async (key) => { - await gone.store.remove("c-1"); + await removeKeys(gone.store, "c-1"); return plain.wrap(key); }, }); @@ -230,13 +244,13 @@ describe("createKeyRotation", () => { const { store, clock, rotation } = await setUp({ ...wrapper, wrap: async (key) => { - await store.remove("c-1"); + await removeKeys(store, "c-1"); return wrapper.wrap(key); }, }); await addUnder(store, "old", "c-1"); expect(await rotation.sweep(clock.now())).toBe(0); - expect(await store.get("c-1")).toBeNull(); + expect(await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); }); it("holds a key that failed to rewrap, longer after each failure, while the others go on", async () => { @@ -248,7 +262,7 @@ describe("createKeyRotation", () => { if ( failing && args.wrapped.length > 0 && - (await store.get("c-1"))?.key === toText(args.wrapped) + (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY))?.key === toText(args.wrapped) ) { throw new Error("unwrap failed"); } @@ -336,7 +350,8 @@ describe("createKeyRotation", () => { const { store, clock, rotation } = await setUp({ ...wrapper, wrap: async (key) => { - if ((await store.get("a-held"))?.kid === "old" && failing) throw new Error("wrap failed"); + if ((await store.get("a-held", DEFAULT_PERSONAL_CATEGORY))?.kid === "old" && failing) + throw new Error("wrap failed"); return wrapper.wrap(key); }, }); @@ -784,13 +799,13 @@ describe("the key rotation across instances", () => { expect(await rotation.sweep(clock.now())).toBe(1); expect(await rotation.sweep(clock.now())).toBe(0); expect(await rotation.nextDueAt()).toBeNull(); - await store.remove("c-2"); - await store.add("c-2", held); + await store.remove("c-2", held.fingerprint); + await store.add("c-2", "default", held); await store.hold("c-2", held, { until: later(HOLD_MAX_MS), attempts: 7 }); for (const { subscriber } of await checkpoints.list()) await checkpoints.remove(subscriber); await rotation.restored(true); expect(await store.listWrappedUnder({ kids: ["old"], limit: 10 })).toEqual([ - { person: "c-2", key: held, hold: null }, + { person: "c-2", category: "default", key: held, hold: null }, ]); expect(await rotation.nextDueAt()).toEqual(clock.now()); expect(await rotation.sweep(clock.now())).toBe(1); diff --git a/packages/core/src/kernel/personal-data/witness.test.ts b/packages/core/src/kernel/personal-data/witness.test.ts index 3e0cdc38..91c51720 100644 --- a/packages/core/src/kernel/personal-data/witness.test.ts +++ b/packages/core/src/kernel/personal-data/witness.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from "vitest"; +import { DEFAULT_PERSONAL_CATEGORY as D } from "../../adapter/storage/personal-key-store.ts"; import { createFixedClock } from "../../contracts/clock.ts"; import { createKeptPersonalKeyStore } from "../../memory/personal-key-store.ts"; import type { PersonalKey } from "./crypto.ts"; @@ -17,24 +18,29 @@ describe("a witnessed key wrapping", () => { onForgetLedger: () => {}, }, ); - const keys = wrapping.over(createKeptPersonalKeyStore().store); - expect(await keys.held("c-1")).toBeNull(); - const created = await keys.create("c-1"); - const held = await keys.held("c-1"); - const stored = await keys.get("c-1"); + const keys = wrapping.over( + createKeptPersonalKeyStore({ restricted: () => false, lastActiveAt: () => null }).store, + ); + expect(await keys.held("c-1", D)).toBeNull(); + expect(await keys.opened("c-1", "f-none")).toBeNull(); + const created = await keys.create("c-1", D); + const held = await keys.held("c-1", D); + const stored = await keys.get("c-1", D); if (stored === null) throw new Error("no key stored"); const opened = await keys.open("c-1", stored); - expect([held?.fingerprint, opened.fingerprint]).toEqual([ + const found = await keys.opened("c-1", created.fingerprint); + const all = await keys.heldAll("c-1"); + expect([held, opened, found, all[0]?.key].map((key) => key?.fingerprint)).toEqual([ + created.fingerprint, + created.fingerprint, created.fingerprint, created.fingerprint, ]); - expect(told.map(([person, key]) => [person, key.fingerprint])).toEqual([ - ["c-1", created.fingerprint], - ["c-1", created.fingerprint], - ["c-1", created.fingerprint], - ]); + expect(told.map(([person, key]) => [person, key.fingerprint])).toEqual( + Array.from({ length: 5 }, () => ["c-1", created.fingerprint]), + ); expect(told[0]?.[1]).toBe(created); - expect(await keys.remove("c-1")).toBe(true); + expect(await keys.remove("c-1", created.fingerprint)).toBe(true); wrapping.close(); }); }); diff --git a/packages/core/src/kernel/personal-data/witness.ts b/packages/core/src/kernel/personal-data/witness.ts index 0762e86a..08d14366 100644 --- a/packages/core/src/kernel/personal-data/witness.ts +++ b/packages/core/src/kernel/personal-data/witness.ts @@ -15,7 +15,8 @@ export interface WitnessedWrappingFunction { (wrapping: PersonalKeyWrapping, witness: PersonalDataWitness): PersonalKeyWrapping; } -// Every key the kernel seals or opens with comes out of `held`, `open` or `create`. +// Every key the kernel seals or opens with comes out of `held`, `opened`, `heldAll`, `open` or +// `create`. export const witnessedWrapping: WitnessedWrappingFunction = (wrapping, witness) => { const seen = (person: string, key: Key): Key => { if (key !== null) witness.onPersonalKey(person, key); @@ -27,9 +28,15 @@ export const witnessedWrapping: WitnessedWrappingFunction = (wrapping, witness) const over = wrapping.over(keys); return { ...over, - held: async (person) => seen(person, await over.held(person)), + held: async (person, category) => seen(person, await over.held(person, category)), + opened: async (person, fingerprint) => seen(person, await over.opened(person, fingerprint)), + heldAll: async (person) => { + const all = await over.heldAll(person); + for (const { key } of all) seen(person, key); + return all; + }, open: async (person, stored) => seen(person, await over.open(person, stored)), - create: async (person) => seen(person, await over.create(person)), + create: async (person, category) => seen(person, await over.create(person, category)), }; }, }; diff --git a/packages/core/src/kernel/personal-data/wrapped-keys.test.ts b/packages/core/src/kernel/personal-data/wrapped-keys.test.ts index f4661f80..6dd1b5f3 100644 --- a/packages/core/src/kernel/personal-data/wrapped-keys.test.ts +++ b/packages/core/src/kernel/personal-data/wrapped-keys.test.ts @@ -1,5 +1,8 @@ import { describe, expect, it, vi } from "vitest"; -import type { PersonalKeyStore } from "../../adapter/storage/personal-key-store.ts"; +import { + DEFAULT_PERSONAL_CATEGORY, + type PersonalKeyStore, +} from "../../adapter/storage/personal-key-store.ts"; import { createFixedClock } from "../../contracts/clock.ts"; import { ConfigurationError } from "../../contracts/errors.ts"; import { memory } from "../../memory/index.ts"; @@ -9,6 +12,7 @@ import { cryptoKeyWrapperOver, defineKeyWrapper, type KeyWrapper } from "./key-w import { buildPersonalKeyWrapping, createPersonalKeyWrapping, + inSealingOrder, UNWRAPPED_KEY_TTL_MS, UNWRAPPED_KEYS_KEPT, } from "./wrapped-keys.ts"; @@ -66,48 +70,139 @@ const fingerprintOf = async (key: Promise): Promise { if (one === null || other === null) throw new Error("expected two keys"); expect(other.fingerprint).toBe(one.fingerprint); - const sealed = await seal({ key: one, person: "c-1", value: "ana@example.com" }); + const sealed = await seal({ + key: one, + person: "c-1", + category: DEFAULT_PERSONAL_CATEGORY, + value: "ana@example.com", + }); expect(await unseal({ key: other, sealed })).toBe("ana@example.com"); }; describe("createPersonalKeyWrapping", () => { + it("hands every call it does not wrap straight to the store, with what it answered", async () => { + const calls: unknown[][] = []; + const store = new Proxy({} as PersonalKeyStore, { + get: + (_, name) => + (...args: unknown[]) => { + calls.push([name, ...args]); + return Promise.resolve(`${String(name)} answer`); + }, + }); + const keys = wrappingOf(null).over(store); + const passed: [keyof PersonalKeyStore, unknown[]][] = [ + ["retain", ["c-1", "f-1", { since: "s", lastActiveAt: "a" }]], + ["touch", ["c-1", "a"]], + ["listDue", [{ category: "x", limit: 1 }]], + ["holdDue", ["c-1", "f-1", "a", { until: "u", attempts: 1 }]], + ["nextDueHoldEnd", [{ category: "x", after: new Date(0) }]], + ["listRetained", [{ except: [], limit: 1 }]], + ["nextRetainedHoldEnd", [{ except: [], after: new Date(0) }]], + ["countDue", [{ category: "x" }]], + ["countRetained", [{ except: ["y"] }]], + ["releaseDueHolds", []], + ["listWrappedUnder", [{ kids: [null], limit: 1 }]], + ["hold", ["c-1", { key: "k", fingerprint: "f-1", kid: null }, { until: "u", attempts: 1 }]], + ["nextHoldEnd", [{ kids: [null], after: new Date(0) }]], + ["countWrappedUnder", [{ kids: [null] }]], + ["releaseHolds", []], + ]; + for (const [name, args] of passed) { + const call = keys[name] as (...args: unknown[]) => Promise; + expect(await call(...args)).toBe(`${name} answer`); + expect(calls.at(-1)).toEqual([name, ...args]); + } + }); + it("stores a new key wrapped under the wrapper's kid, named by its fingerprint, without unwrapping it", async () => { const store = await keyStore(); const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper).over(store); - const key = await keys.create("c-1"); - const stored = await store.get("c-1"); + const key = await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); + const stored = await store.get("c-1", DEFAULT_PERSONAL_CATEGORY); expect(stored).toEqual({ key: expect.any(String), fingerprint: key.fingerprint, kid: "kid-1" }); expect(wrapper.unwrapped).toEqual([]); - await expectSameKey(key, await wrappingOf(wrapper).over(store).held("c-1")); + await expectSameKey( + key, + await wrappingOf(wrapper).over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY), + ); expect(wrapper.unwrapped).toEqual(["kid-1"]); - expect(await keys.get("c-1")).toEqual(stored); + expect(await keys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toEqual(stored); }); it("answers a create with the key another writer stored first, unwrapped", async () => { const store = await keyStore(); const wrapper = fakeWrapper("kid-1"); - const first = await wrappingOf(wrapper).over(store).create("c-1"); - await expectSameKey(first, await wrappingOf(wrapper).over(store).create("c-1")); + const first = await wrappingOf(wrapper).over(store).create("c-1", DEFAULT_PERSONAL_CATEGORY); + await expectSameKey( + first, + await wrappingOf(wrapper).over(store).create("c-1", DEFAULT_PERSONAL_CATEGORY), + ); expect(wrapper.unwrapped).toEqual(["kid-1"]); }); it("opens the key a row read for the person holds", async () => { const store = await keyStore(); const wrapper = fakeWrapper("kid-1"); - const key = await wrappingOf(wrapper).over(store).create("c-1"); + const key = await wrappingOf(wrapper).over(store).create("c-1", DEFAULT_PERSONAL_CATEGORY); const keys = wrappingOf(wrapper).over(store); - const stored = await keys.get("c-1"); + const stored = await keys.get("c-1", DEFAULT_PERSONAL_CATEGORY); if (stored === null) throw new Error("expected a key"); await expectSameKey(key, await keys.open("c-1", stored)); - expect(await keys.held("c-2")).toBeNull(); + expect(await keys.held("c-2", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); + }); + + it("keeps one key in force per category, and opens a key by fingerprint, in force or retained", async () => { + const store = await keyStore(); + const keys = wrappingOf(fakeWrapper("kid-1")).over(store); + const profile = await keys.create("c-1", "profile"); + const invoices = await keys.create("c-1", "invoices"); + expect(invoices.fingerprint).not.toBe(profile.fingerprint); + await expectSameKey(profile, await keys.held("c-1", "profile")); + expect( + await store.retain("c-1", invoices.fingerprint, { + since: "2026-01-01T00:00:00.000Z", + lastActiveAt: "2025-12-01T00:00:00.000Z", + }), + ).toBe(true); + expect(await keys.held("c-1", "invoices")).toBeNull(); + const retained = await keys.opened("c-1", invoices.fingerprint); + expect(retained?.retained).toBe(true); + await expectSameKey(invoices, retained); + expect((await keys.opened("c-1", profile.fingerprint))?.retained).toBe(false); + expect(await keys.opened("c-2", profile.fingerprint)).toBeNull(); + }); + + it("opens every key the person holds, in force or retained, with its category", async () => { + const store = await keyStore(); + const keys = wrappingOf(null).over(store); + const profile = await keys.create("c-1", "profile"); + const invoices = await keys.create("c-1", "invoices"); + await keys.create("c-2", "profile"); + await store.retain("c-1", invoices.fingerprint, { + since: "2026-01-01T00:00:00.000Z", + lastActiveAt: "2025-12-01T00:00:00.000Z", + }); + const all = await keys.heldAll("c-1"); + expect(all.map(({ category, key }) => [category, key.fingerprint, key.retained])).toEqual( + expect.arrayContaining([ + ["profile", profile.fingerprint, false], + ["invoices", invoices.fingerprint, true], + ]), + ); + expect(all).toHaveLength(2); + await expectSameKey(invoices, all.find(({ category }) => category === "invoices")?.key ?? null); + expect(await keys.heldAll("c-3")).toEqual([]); }); it("holds every key as one that can never be exported: new, read, wrapped or not", async () => { for (const wrapper of [fakeWrapper("kid-1"), null]) { const store = await keyStore(); - const created = await wrappingOf(wrapper).over(store).create("c-1"); - const read = await wrappingOf(wrapper).over(store).held("c-1"); + const created = await wrappingOf(wrapper) + .over(store) + .create("c-1", DEFAULT_PERSONAL_CATEGORY); + const read = await wrappingOf(wrapper).over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY); for (const key of [created, read]) { expect(key?.cryptoKey.extractable).toBe(false); expect(key?.cryptoKey.usages.toSorted()).toEqual(["decrypt", "encrypt"]); @@ -121,11 +216,13 @@ describe("createPersonalKeyWrapping", () => { it("zeroes every raw key it hands to a wrapper of the app's own, or is handed back", async () => { const store = await keyStore(); const wrapper = fakeWrapper("kid-1", ["kid-0"]); - await wrappingOf(fakeWrapper("kid-0")).over(store).create("c-1"); + await wrappingOf(fakeWrapper("kid-0")).over(store).create("c-1", DEFAULT_PERSONAL_CATEGORY); const wrapping = wrappingOf(wrapper); - await wrapping.over(store).create("c-2"); - await wrapping.over(store).held("c-1"); - await wrapping.wrapped((await store.get("c-1")) ?? (await newStoredKey())); + await wrapping.over(store).create("c-2", DEFAULT_PERSONAL_CATEGORY); + await wrapping.over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY); + await wrapping.wrapped( + (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)) ?? (await newStoredKey()), + ); expect(wrapper.raw.length).toBeGreaterThanOrEqual(4); for (const raw of wrapper.raw) expect(raw).toEqual(new Uint8Array(32)); }); @@ -134,40 +231,43 @@ describe("createPersonalKeyWrapping", () => { const store = await keyStore(); const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper).over(store); - const key = await keys.create("c-1"); - await expectSameKey(key, await keys.held("c-1")); - await expectSameKey(key, await keys.held("c-1")); + const key = await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); + await expectSameKey(key, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); + await expectSameKey(key, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); expect(wrapper.unwrapped).toEqual([]); // Another instance forgets the person: the cache still holds the key, the row is gone. - expect(await store.remove("c-1")).toBe(true); - expect(await keys.held("c-1")).toBeNull(); + expect(await store.remove("c-1", key.fingerprint)).toBe(true); + expect(await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); }); it("drops the key of a person it forgets", async () => { const store = await keyStore(); const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper).over(store); - const key = await keys.create("c-1"); - const stored = (await store.get("c-1")) ?? (await newStoredKey()); - expect(await keys.remove("c-1")).toBe(true); + const key = await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); + const stored = (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)) ?? (await newStoredKey()); + expect(await keys.remove("c-1", key.fingerprint)).toBe(true); // The same row put back, as a restore would: unwrapped again, so the cache had let it go. - await store.add("c-1", stored); - await expectSameKey(key, await keys.held("c-1")); + await store.add("c-1", DEFAULT_PERSONAL_CATEGORY, stored); + await expectSameKey(key, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); expect(wrapper.unwrapped).toEqual(["kid-1"]); - expect(await keys.remove("c-2")).toBe(false); + expect(await keys.remove("c-2", key.fingerprint)).toBe(false); // Read, not created, by this instance: the same. - expect(await keys.remove("c-1")).toBe(true); - await store.add("c-1", stored); - await expectSameKey(key, await keys.held("c-1")); + expect(await keys.remove("c-1", key.fingerprint)).toBe(true); + await store.add("c-1", DEFAULT_PERSONAL_CATEGORY, stored); + await expectSameKey(key, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); expect(wrapper.unwrapped).toEqual(["kid-1", "kid-1"]); }); it("unwraps once for reads made at the same time", async () => { const store = await keyStore(); const wrapper = fakeWrapper("kid-1"); - const key = await wrappingOf(wrapper).over(store).create("c-1"); + const key = await wrappingOf(wrapper).over(store).create("c-1", DEFAULT_PERSONAL_CATEGORY); const keys = wrappingOf(wrapper).over(store); - const [one, other] = await Promise.all([keys.held("c-1"), keys.held("c-1")]); + const [one, other] = await Promise.all([ + keys.held("c-1", DEFAULT_PERSONAL_CATEGORY), + keys.held("c-1", DEFAULT_PERSONAL_CATEGORY), + ]); await expectSameKey(key, one); expect(other?.cryptoKey).toBe(one?.cryptoKey); expect(wrapper.unwrapped).toEqual(["kid-1"]); @@ -175,7 +275,9 @@ describe("createPersonalKeyWrapping", () => { it("unwraps again a key whose unwrapping failed", async () => { const store = await keyStore(); - const key = await wrappingOf(fakeWrapper("kid-1")).over(store).create("c-1"); + const key = await wrappingOf(fakeWrapper("kid-1")) + .over(store) + .create("c-1", DEFAULT_PERSONAL_CATEGORY); let fail = true; const flaky: KeyWrapper = { ...fakeWrapper("kid-1"), @@ -185,9 +287,9 @@ describe("createPersonalKeyWrapping", () => { }, }; const keys = wrappingOf(flaky).over(store); - await expect(keys.held("c-1")).rejects.toThrow("KMS unavailable"); + await expect(keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)).rejects.toThrow("KMS unavailable"); fail = false; - await expectSameKey(key, await keys.held("c-1")); + await expectSameKey(key, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); }); it("keeps UNWRAPPED_KEYS_KEPT keys by default", () => { @@ -199,21 +301,21 @@ describe("createPersonalKeyWrapping", () => { const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper, { keep: 3 }).over(store); const persons = Array.from({ length: 4 }, (_, index) => `c-${index}`); - for (const person of persons) await keys.create(person); + for (const person of persons) await keys.create(person, DEFAULT_PERSONAL_CATEGORY); const reads = () => wrapper.unwrapped.length; let before = reads(); // The first was the oldest, so it went out when the last came in. - await keys.held("c-0"); + await keys.held("c-0", DEFAULT_PERSONAL_CATEGORY); expect(reads()).toBe(before + 1); // Reading `c-2` makes it recent, so the next eviction takes `c-3`, not it. before = reads(); - await keys.held("c-2"); + await keys.held("c-2", DEFAULT_PERSONAL_CATEGORY); expect(reads()).toBe(before); - await keys.create("c-new"); + await keys.create("c-new", DEFAULT_PERSONAL_CATEGORY); before = reads(); - await keys.held("c-2"); + await keys.held("c-2", DEFAULT_PERSONAL_CATEGORY); expect(reads()).toBe(before); - await keys.held("c-3"); + await keys.held("c-3", DEFAULT_PERSONAL_CATEGORY); expect(reads()).toBe(before + 1); }); @@ -223,12 +325,12 @@ describe("createPersonalKeyWrapping", () => { const timed = createFixedClock(); const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper, { clock: timed }).over(store); - await keys.create("c-1"); + await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); timed.advance(UNWRAPPED_KEY_TTL_MS - 1); - await keys.held("c-1"); + await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(0); timed.advance(1); - await keys.held("c-1"); + await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(1); }); @@ -237,21 +339,21 @@ describe("createPersonalKeyWrapping", () => { const timed = createFixedClock(); const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper, { clock: timed, ttlMs: 1_000 }).over(store); - const first = await keys.create("c-1"); + const first = await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); timed.advance(600); - await keys.create("c-2"); + await keys.create("c-2", DEFAULT_PERSONAL_CATEGORY); expect(timed.pending()).toBe(1); timed.advance(399); - await expectSameKey(first, await keys.held("c-1")); + await expectSameKey(first, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); expect(wrapper.unwrapped).toHaveLength(0); // Nothing reads: the timer drops the first key when its time is up, then the second. timed.advance(1); expect(timed.pending()).toBe(1); timed.advance(600); expect(timed.pending()).toBe(0); - await expectSameKey(first, await keys.held("c-1")); + await expectSameKey(first, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); expect(wrapper.unwrapped).toHaveLength(1); - expect(await keys.remove("c-1")).toBe(true); + expect(await keys.remove("c-1", first.fingerprint)).toBe(true); timed.advance(1_000); expect(timed.pending()).toBe(0); }); @@ -261,12 +363,12 @@ describe("createPersonalKeyWrapping", () => { const wrapper = fakeWrapper("kid-1"); const late: typeof clock = { ...createFixedClock(), after: () => () => undefined }; const keys = wrappingOf(wrapper, { clock: late, ttlMs: 1_000 }).over(store); - const key = await keys.create("c-1"); + const key = await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); late.advance(999); - await keys.held("c-1"); + await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(0); late.advance(1); - await expectSameKey(key, await keys.held("c-1")); + await expectSameKey(key, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); expect(wrapper.unwrapped).toHaveLength(1); }); @@ -275,23 +377,25 @@ describe("createPersonalKeyWrapping", () => { const wrapper = fakeWrapper("kid-1"); const late: typeof clock = { ...createFixedClock(), after: () => () => undefined }; const keys = wrappingOf(wrapper, { clock: late, keep: 2, ttlMs: 1_000 }).over(store); - await keys.create("c-1"); + await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); late.advance(500); - await keys.create("c-2"); + await keys.create("c-2", DEFAULT_PERSONAL_CATEGORY); late.advance(500); - await keys.held("c-1"); + await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(1); late.advance(100); - await keys.create("c-3"); - await keys.held("c-1"); + await keys.create("c-3", DEFAULT_PERSONAL_CATEGORY); + await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(1); - await keys.held("c-2"); + await keys.held("c-2", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(2); }); it("keeps a key unwrapped again when an older unwrapping of it fails late", async () => { const store = await keyStore(); - const key = await wrappingOf(fakeWrapper("kid-1")).over(store).create("c-1"); + const key = await wrappingOf(fakeWrapper("kid-1")) + .over(store) + .create("c-1", DEFAULT_PERSONAL_CATEGORY); let fail: (error: Error) => void = () => undefined; let calls = 0; const slow: KeyWrapper = { @@ -308,13 +412,13 @@ describe("createPersonalKeyWrapping", () => { }; const late: typeof clock = { ...createFixedClock(), after: () => () => undefined }; const keys = wrappingOf(slow, { clock: late, ttlMs: 1_000 }).over(store); - const first = keys.held("c-1"); + const first = keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); await new Promise((resolve) => setImmediate(resolve)); late.advance(1_000); - await expectSameKey(key, await keys.held("c-1")); + await expectSameKey(key, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); fail(new Error("KMS timed out")); await expect(first).rejects.toThrow("KMS timed out"); - await expectSameKey(key, await keys.held("c-1")); + await expectSameKey(key, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); expect(calls).toBe(2); }); @@ -323,17 +427,17 @@ describe("createPersonalKeyWrapping", () => { const wrapper = fakeWrapper("kid-1", ["kid-0"]); const wrapping = wrappingOf(wrapper); const keys = wrapping.over(store); - await keys.create("c-1"); - const stored = (await store.get("c-1")) ?? (await newStoredKey()); + await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); + const stored = (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)) ?? (await newStoredKey()); await wrapping.wrapped(stored); await wrapping.wrapped(stored); expect(wrapper.unwrapped).toHaveLength(2); - await keys.held("c-1"); + await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(2); wrapping.drop("c-1"); wrapping.drop("c-9"); - await keys.held("c-1"); - await keys.held("c-1"); + await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); + await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(3); }); @@ -342,9 +446,9 @@ describe("createPersonalKeyWrapping", () => { const timed = createFixedClock(new Date("2026-01-01T01:00:00Z")); const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper, { clock: timed, ttlMs: 1_000 }).over(store); - await keys.create("c-1"); + await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); timed.set(new Date("2026-01-01T00:00:00Z")); - await keys.held("c-1"); + await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(1); }); @@ -352,16 +456,16 @@ describe("createPersonalKeyWrapping", () => { const store = await keyStore(); const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper, { keep: 2 }).over(store); - await keys.create("c-1"); + const old = await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); // Forgotten on another instance, then keyed anew here: the old key is still kept. - await store.remove("c-1"); - const renewed = await keys.create("c-1"); - const stored = (await store.get("c-1")) ?? (await newStoredKey()); + await store.remove("c-1", old.fingerprint); + const renewed = await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); + const stored = (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)) ?? (await newStoredKey()); // The old key goes out, oldest; the person's new one must stay theirs. - await keys.create("c-2"); - expect(await keys.remove("c-1")).toBe(true); - await store.add("c-1", stored); - await expectSameKey(renewed, await keys.held("c-1")); + await keys.create("c-2", DEFAULT_PERSONAL_CATEGORY); + expect(await keys.remove("c-1", renewed.fingerprint)).toBe(true); + await store.add("c-1", DEFAULT_PERSONAL_CATEGORY, stored); + await expectSameKey(renewed, await keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)); expect(wrapper.unwrapped).toEqual(["kid-1"]); }); @@ -369,10 +473,10 @@ describe("createPersonalKeyWrapping", () => { const store = await keyStore(); const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper).over(store); - await keys.create("c-1"); - const kept = await keys.create("c-2"); - await keys.remove("c-1"); - await expectSameKey(kept, await keys.held("c-2")); + const forgotten = await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); + const kept = await keys.create("c-2", DEFAULT_PERSONAL_CATEGORY); + await keys.remove("c-1", forgotten.fingerprint); + await expectSameKey(kept, await keys.held("c-2", DEFAULT_PERSONAL_CATEGORY)); expect(wrapper.unwrapped).toHaveLength(0); }); @@ -386,7 +490,9 @@ describe("createPersonalKeyWrapping", () => { return () => undefined; }, }; - await wrappingOf(fakeWrapper("kid-1"), { clock: watched }).over(store).create("c-1"); + await wrappingOf(fakeWrapper("kid-1"), { clock: watched }) + .over(store) + .create("c-1", DEFAULT_PERSONAL_CATEGORY); expect(options).toEqual([{ background: true }]); }); @@ -395,46 +501,53 @@ describe("createPersonalKeyWrapping", () => { const timed = createFixedClock(); const wrapper = fakeWrapper("kid-1"); const wrapping = wrappingOf(wrapper, { clock: timed }); - await wrapping.over(store).create("c-1"); + await wrapping.over(store).create("c-1", DEFAULT_PERSONAL_CATEGORY); expect(timed.pending()).toBe(1); wrapping.close(); expect(timed.pending()).toBe(0); - await wrapping.over(store).held("c-1"); + await wrapping.over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY); expect(wrapper.unwrapped).toHaveLength(1); }); it("opens keys kept unwrapped, from before a wrapper was set, without the wrapper", async () => { const store = await keyStore(); const stored = await newStoredKey(); - await store.add("c-1", stored); + await store.add("c-1", DEFAULT_PERSONAL_CATEGORY, stored); const wrapper = fakeWrapper("kid-1"); - expect(await fingerprintOf(wrappingOf(wrapper).over(store).held("c-1"))).toBe( - stored.fingerprint, - ); + expect( + await fingerprintOf(wrappingOf(wrapper).over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY)), + ).toBe(stored.fingerprint); expect(wrapper.unwrapped).toEqual([]); }); it("unwraps a key under a previous kid, and rewraps it under the current one", async () => { const store = await keyStore(); - const key = await wrappingOf(fakeWrapper("old")).over(store).create("c-1"); - const stored = (await store.get("c-1")) ?? (await newStoredKey()); + const key = await wrappingOf(fakeWrapper("old")) + .over(store) + .create("c-1", DEFAULT_PERSONAL_CATEGORY); + const stored = (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)) ?? (await newStoredKey()); const wrapping = wrappingOf(fakeWrapper("current", ["old"])); - await expectSameKey(key, await wrapping.over(store).held("c-1")); + await expectSameKey(key, await wrapping.over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY)); const rewrapped = await wrapping.wrapped(stored); expect(rewrapped.kid).toBe("current"); expect(rewrapped.key).not.toBe(stored.key); const moved = await keyStore(); - await moved.add("c-1", { ...rewrapped, fingerprint: key.fingerprint }); - await expectSameKey(key, await wrapping.over(moved).held("c-1")); + await moved.add("c-1", DEFAULT_PERSONAL_CATEGORY, { + ...rewrapped, + fingerprint: key.fingerprint, + }); + await expectSameKey(key, await wrapping.over(moved).held("c-1", DEFAULT_PERSONAL_CATEGORY)); }); it("imports a new key once without a wrapper, never reading it back from the store", async () => { const store = await keyStore(); const imported = vi.spyOn(crypto.subtle, "importKey"); try { - const key = await wrappingOf(null).over(store).create("c-1"); + const key = await wrappingOf(null).over(store).create("c-1", DEFAULT_PERSONAL_CATEGORY); expect(imported).toHaveBeenCalledTimes(1); - expect(key.fingerprint).toBe((await store.get("c-1"))?.fingerprint); + expect(key.fingerprint).toBe( + (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY))?.fingerprint, + ); } finally { imported.mockRestore(); } @@ -443,21 +556,26 @@ describe("createPersonalKeyWrapping", () => { it("keeps keys as they are without a wrapper", async () => { const store = await keyStore(); const wrapping = wrappingOf(null); - const key = await wrapping.over(store).create("c-1"); - const stored = await store.get("c-1"); + const key = await wrapping.over(store).create("c-1", DEFAULT_PERSONAL_CATEGORY); + const stored = await store.get("c-1", DEFAULT_PERSONAL_CATEGORY); expect(stored).toEqual({ key: expect.any(String), fingerprint: key.fingerprint, kid: null }); expect(fromBase64(stored?.key ?? "")).toHaveLength(32); expect(await wrapping.wrapped(stored ?? (await newStoredKey()))).toEqual({ key: stored?.key, kid: null, }); - await expectSameKey(key, await wrappingOf(null).over(store).held("c-1")); + await expectSameKey( + key, + await wrappingOf(null).over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY), + ); }); it("refuses a wrapped key without a wrapper, under an unknown kid, or unwrapping to another key", async () => { const store = await keyStore(); - await wrappingOf(fakeWrapper("kid-1")).over(store).create("c-1"); - await expect(wrappingOf(null).over(store).held("c-1")).rejects.toThrow( + await wrappingOf(fakeWrapper("kid-1")).over(store).create("c-1", DEFAULT_PERSONAL_CATEGORY); + await expect( + wrappingOf(null).over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY), + ).rejects.toThrow( new ConfigurationError( "A personal key is wrapped under kid-1, and runtime.personalData.keyWrapper is not set", ), @@ -465,28 +583,37 @@ describe("createPersonalKeyWrapping", () => { await expect( wrappingOf(fakeWrapper("kid-2", ["kid-3"])) .over(store) - .held("c-1"), + .held("c-1", DEFAULT_PERSONAL_CATEGORY), ).rejects.toThrow( new ConfigurationError( "A personal key is wrapped under kid-1, which is neither the key wrapper's key nor one of its previous ones", ), ); const liar: KeyWrapper = { ...fakeWrapper("kid-1"), unwrap: async () => new Uint8Array(32) }; - await expect(wrappingOf(liar).over(store).held("c-1")).rejects.toThrow( + await expect( + wrappingOf(liar).over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY), + ).rejects.toThrow( new ConfigurationError( "A personal key unwrapped under kid-1 does not match its fingerprint: the key wrapper returned another key, or the database was written before fingerprints were key check values and must be created anew", ), ); await expect( - wrappingOf(liar).wrapped((await store.get("c-1")) ?? (await newStoredKey())), + wrappingOf(liar).wrapped( + (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)) ?? (await newStoredKey()), + ), ).rejects.toThrow("does not match its fingerprint"); }); it("refuses a key kept unwrapped that does not match its fingerprint", async () => { const store = await keyStore(); const [stored, other] = await Promise.all([newStoredKey(), newStoredKey()]); - await store.add("c-1", { ...stored, fingerprint: other.fingerprint }); - await expect(wrappingOf(fakeWrapper("kid-1")).over(store).held("c-1")).rejects.toThrow( + await store.add("c-1", DEFAULT_PERSONAL_CATEGORY, { + ...stored, + fingerprint: other.fingerprint, + }); + await expect( + wrappingOf(fakeWrapper("kid-1")).over(store).held("c-1", DEFAULT_PERSONAL_CATEGORY), + ).rejects.toThrow( new ConfigurationError( "A personal key kept unwrapped does not match its fingerprint: the key was altered in the store, or the database was written before fingerprints were key check values and must be created anew", ), @@ -496,10 +623,10 @@ describe("createPersonalKeyWrapping", () => { it("refuses a key rewrapped under a kid it does not know, however recently it kept the key", async () => { const store = await keyStore(); const keys = wrappingOf(fakeWrapper("kid-1")).over(store); - await keys.create("c-1"); - const stored = (await store.get("c-1")) ?? (await newStoredKey()); + await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); + const stored = (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)) ?? (await newStoredKey()); await store.rewrap("c-1", stored, { key: stored.key, kid: "kid-9" }); - await expect(keys.held("c-1")).rejects.toThrow( + await expect(keys.held("c-1", DEFAULT_PERSONAL_CATEGORY)).rejects.toThrow( "A personal key is wrapped under kid-9, which is neither the key wrapper's key nor one of its previous ones", ); }); @@ -509,31 +636,37 @@ describe("createPersonalKeyWrapping", () => { const wrapper = fakeWrapper("kid-1"); const keys = wrappingOf(wrapper).over(store); const plain = await newStoredKey(); - expect(await keys.add("c-1", plain)).toMatchObject({ + expect(await keys.add("c-1", DEFAULT_PERSONAL_CATEGORY, plain)).toMatchObject({ fingerprint: plain.fingerprint, kid: "kid-1", }); - expect((await store.get("c-1"))?.key).not.toBe(plain.key); - expect(await fingerprintOf(keys.held("c-1"))).toBe(plain.fingerprint); - const wrappedRow = (await store.get("c-1")) ?? plain; + expect((await store.get("c-1", DEFAULT_PERSONAL_CATEGORY))?.key).not.toBe(plain.key); + expect(await fingerprintOf(keys.held("c-1", DEFAULT_PERSONAL_CATEGORY))).toBe( + plain.fingerprint, + ); + const wrappedRow = (await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)) ?? plain; const unwraps = wrapper.unwrapped.length; - expect(await keys.add("c-2", wrappedRow)).toEqual(wrappedRow); + expect(await keys.add("c-2", DEFAULT_PERSONAL_CATEGORY, wrappedRow)).toEqual(wrappedRow); expect(wrapper.unwrapped).toHaveLength(unwraps); - expect(await wrappingOf(null).over(store).add("c-3", plain)).toEqual(plain); + expect(await wrappingOf(null).over(store).add("c-3", DEFAULT_PERSONAL_CATEGORY, plain)).toEqual( + plain, + ); }); it("passes reading, adding, forgetting, listing and rewrapping through to the store", async () => { const store = await keyStore(); const keys = wrappingOf(fakeWrapper("kid-1")).over(store); - const key = await keys.create("c-1"); + const key = await keys.create("c-1", DEFAULT_PERSONAL_CATEGORY); const [row] = await keys.listWrappedUnder({ kids: ["kid-1"], limit: 10 }); expect(row?.person).toBe("c-1"); const stored = row?.key ?? (await newStoredKey()); - expect(await keys.get("c-1")).toEqual(stored); + expect(await keys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toEqual(stored); const other = await wrappingOf(fakeWrapper("kid-1")) .over(await keyStore()) - .create("c-2"); - expect((await keys.add("c-2", stored)).fingerprint).toBe(key.fingerprint); + .create("c-2", DEFAULT_PERSONAL_CATEGORY); + expect((await keys.add("c-2", DEFAULT_PERSONAL_CATEGORY, stored)).fingerprint).toBe( + key.fingerprint, + ); expect(other.fingerprint).not.toBe(key.fingerprint); const hold = { until: "2026-01-01T00:00:30.000Z", attempts: 1 }; await keys.hold("c-1", stored, hold); @@ -542,13 +675,13 @@ describe("createPersonalKeyWrapping", () => { new Date(hold.until), ); expect(await keys.rewrap("c-1", stored, { key: "other", kid: "kid-2" })).toBe(true); - expect(await store.get("c-1")).toEqual({ + expect(await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toEqual({ key: "other", fingerprint: key.fingerprint, kid: "kid-2", }); - expect(await keys.remove("c-1")).toBe(true); - expect(await store.get("c-1")).toBeNull(); + expect(await keys.remove("c-1", key.fingerprint)).toBe(true); + expect(await store.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); }); }); @@ -570,3 +703,21 @@ describe("buildPersonalKeyWrapping", () => { ).toBeNull(); }); }); + +describe("inSealingOrder", () => { + it("orders keys by category, then by fingerprint, whatever order they came in", () => { + const key = (category: string, fingerprint: string) => ({ + category, + key: { fingerprint, cryptoKey: {} as CryptoKey, retained: false }, + }); + const ordered = inSealingOrder([ + key("profile", "b"), + key("default", "z"), + key("profile", "a"), + key("invoices", "m"), + ]); + expect( + ordered.map(({ category, key: { fingerprint } }) => `${category}:${fingerprint}`), + ).toEqual(["default:z", "invoices:m", "profile:a", "profile:b"]); + }); +}); diff --git a/packages/core/src/kernel/personal-data/wrapped-keys.ts b/packages/core/src/kernel/personal-data/wrapped-keys.ts index 2fe96488..1c6db4e4 100644 --- a/packages/core/src/kernel/personal-data/wrapped-keys.ts +++ b/packages/core/src/kernel/personal-data/wrapped-keys.ts @@ -44,19 +44,55 @@ const OLDER_DATABASE = export interface PersonalKeys extends PersonalKeyStore { /** - * The key the person holds, `null` when they hold none. + * The key the person holds in force for the category, `null` when they hold none. */ - held(person: string): Promise; + held(person: string, category: string): Promise; /** - * The key `stored` holds, as `get` read it for the person. + * The key of that fingerprint the person holds, in force or retained, `null` once it is gone. + */ + opened(person: string, fingerprint: string): Promise; + /** + * Every key the person holds, in force or retained, with its category, by fingerprint. + */ + heldAll(person: string): Promise; + /** + * The key `stored` holds, as `get` read it in force for the person. */ open(person: string, stored: StoredPersonalKey): Promise; /** - * Stores a new key for the person unless one is in force, and resolves the one in force. + * Stores a new key for the person and category unless one is in force, and resolves the one in + * force. */ - create(person: string): Promise; + create(person: string, category: string): Promise; +} + +/** + * One of a person's keys and the category it seals. + */ +export interface CategoryKey { + readonly category: string; + readonly key: PersonalKey; } +export interface InSealingOrderFunction { + (keys: readonly CategoryKey[]): readonly CategoryKey[]; +} + +/** + * Keys in the order text is sealed under all of them, by category, then by fingerprint: the same + * keys always nest alike. + */ +export const inSealingOrder: InSealingOrderFunction = (keys) => + [...keys].sort((a, b) => + a.category === b.category + ? a.key.fingerprint < b.key.fingerprint + ? -1 + : 1 + : a.category < b.category + ? -1 + : 1, + ); + export interface PersonalKeyWrapping { /** * `null` without a key wrapper: keys are kept as they are. @@ -68,7 +104,7 @@ export interface PersonalKeyWrapping { */ wrapped(key: StoredPersonalKey): Promise; /** - * Drops the person's key, for a forgetting made on the store as written. + * Drops the person's keys, for a forgetting made on the store as written. */ drop(person: string): void; over(keys: PersonalKeyStore): PersonalKeys; @@ -118,8 +154,8 @@ export const createPersonalKeyWrapping: CreatePersonalKeyWrappingFunction = ({ const kept = new Map(); // Kept first, which, kept as long as each other, is also the first to expire. const keptAt = new Map(); - // The fingerprint kept for each person, so forgetting them drops their key at once. - const byPerson = new Map(); + // The fingerprints kept for each person, so forgetting them drops their keys at once. + const byPerson = new Map>(); const known = new Set(wrapper === null ? [] : [wrapper.kid, ...wrapper.previous]); let cancelExpiry: (() => void) | null = null; @@ -127,7 +163,10 @@ export const createPersonalKeyWrapping: CreatePersonalKeyWrappingFunction = ({ const person = kept.get(fingerprint)?.person; kept.delete(fingerprint); keptAt.delete(fingerprint); - if (person !== undefined && byPerson.get(person) === fingerprint) byPerson.delete(person); + if (person === undefined) return; + const fingerprints = byPerson.get(person); + fingerprints?.delete(fingerprint); + if (fingerprints?.size === 0) byPerson.delete(person); }; // A wall clock set back makes a key look younger than it can be: it is taken as expired. @@ -196,7 +235,7 @@ export const createPersonalKeyWrapping: CreatePersonalKeyWrappingFunction = ({ const created: KeptKey = { key, expiresAt: clock.now().getTime() + ttlMs, person }; kept.set(fingerprint, created); keptAt.set(fingerprint, created); - byPerson.set(person, fingerprint); + byPerson.set(person, (byPerson.get(person) ?? new Set()).add(fingerprint)); key.catch(() => { if (kept.get(fingerprint) === created) drop(fingerprint); }); @@ -205,7 +244,11 @@ export const createPersonalKeyWrapping: CreatePersonalKeyWrappingFunction = ({ armExpiry(); }; - const open = async (person: string, stored: StoredPersonalKey): Promise => { + const open = async ( + person: string, + stored: StoredPersonalKey, + retained = false, + ): Promise => { const { kid, fingerprint } = stored; // Before the cache, so a key rewrapped under a kid this app does not know fails however // recently it was kept. @@ -215,11 +258,11 @@ export const createPersonalKeyWrapping: CreatePersonalKeyWrappingFunction = ({ if (entry === undefined) { const key = unwrap(stored, false); remember(fingerprint, person, key); - return { fingerprint, cryptoKey: await key }; + return { fingerprint, cryptoKey: await key, retained }; } kept.delete(fingerprint); kept.set(fingerprint, entry); - return { fingerprint, cryptoKey: await entry.key }; + return { fingerprint, cryptoKey: await entry.key, retained }; }; // The new key as the store keeps it, and as it is sealed with. Its raw bytes exist only without @@ -231,13 +274,16 @@ export const createPersonalKeyWrapping: CreatePersonalKeyWrappingFunction = ({ return withRawKey(exportable, async (raw) => { const text = toBase64(raw); const cryptoKey = await importRawKey({ raw, extractable: false }); - return { stored: { key: text, fingerprint, kid: null }, key: { fingerprint, cryptoKey } }; + return { + stored: { key: text, fingerprint, kid: null }, + key: { fingerprint, cryptoKey, retained: false }, + }; }); } const { wrapped, key: cryptoKey } = await wrapper.wrap(exportable); return { stored: { key: toBase64(wrapped), fingerprint, kid: wrapper.kid }, - key: { fingerprint, cryptoKey }, + key: { fingerprint, cryptoKey, retained: false }, }; }; @@ -248,8 +294,7 @@ export const createPersonalKeyWrapping: CreatePersonalKeyWrappingFunction = ({ }; const dropPerson = (person: string): void => { - const fingerprint = byPerson.get(person); - if (fingerprint !== undefined) drop(fingerprint); + for (const fingerprint of [...(byPerson.get(person) ?? [])]) drop(fingerprint); }; return { @@ -257,34 +302,59 @@ export const createPersonalKeyWrapping: CreatePersonalKeyWrappingFunction = ({ wrapped, drop: dropPerson, over: (keys) => ({ - get: (person) => keys.get(person), + get: (person, category) => keys.get(person, category), + find: (person, fingerprint) => keys.find(person, fingerprint), + list: (person) => keys.list(person), // Wrapped on the way in, as every key the store keeps. - add: async (person, key) => + add: async (person, category, key) => keys.add( person, + category, key.kid === (wrapper?.kid ?? null) ? key : { ...(await wrapped(key)), fingerprint: key.fingerprint }, ), - held: async (person) => { - const stored = await keys.get(person); + held: async (person, category) => { + const stored = await keys.get(person, category); return stored === null ? null : open(person, stored); }, - open, - create: async (person) => { + opened: async (person, fingerprint) => { + const found = await keys.find(person, fingerprint); + return found === null ? null : open(person, found.key, found.retainedSince !== null); + }, + heldAll: async (person) => + Promise.all( + (await keys.list(person)).map(async (found) => ({ + category: found.category, + key: await open(person, found.key, found.retainedSince !== null), + })), + ), + open: (person, stored) => open(person, stored), + create: async (person, category) => { const fresh = await generate(); - const inForce = await keys.add(person, fresh.stored); + const inForce = await keys.add(person, category, fresh.stored); if (inForce.fingerprint !== fresh.key.fingerprint) return open(person, inForce); remember(fresh.key.fingerprint, person, Promise.resolve(fresh.key.cryptoKey)); return fresh.key; }, // What this instance kept goes with the row; another instance's goes when it expires, and // is never read again meanwhile, since every read starts from the row. - remove: async (person) => { - const removed = await keys.remove(person); - dropPerson(person); + remove: async (person, fingerprint) => { + const removed = await keys.remove(person, fingerprint); + drop(fingerprint); return removed; }, + retain: (person, fingerprint, args) => keys.retain(person, fingerprint, args), + touch: (person, lastActiveAt) => keys.touch(person, lastActiveAt), + listDue: (args) => keys.listDue(args), + holdDue: (person, fingerprint, lastActiveAt, hold) => + keys.holdDue(person, fingerprint, lastActiveAt, hold), + nextDueHoldEnd: (args) => keys.nextDueHoldEnd(args), + listRetained: (args) => keys.listRetained(args), + nextRetainedHoldEnd: (args) => keys.nextRetainedHoldEnd(args), + countDue: (args) => keys.countDue(args), + countRetained: (args) => keys.countRetained(args), + releaseDueHolds: () => keys.releaseDueHolds(), listWrappedUnder: (args) => keys.listWrappedUnder(args), rewrap: (person, from, to) => keys.rewrap(person, from, to), hold: (person, from, hold) => keys.hold(person, from, hold), diff --git a/packages/core/src/kernel/policy/executor.ts b/packages/core/src/kernel/policy/executor.ts index 5866c295..fd9eb579 100644 --- a/packages/core/src/kernel/policy/executor.ts +++ b/packages/core/src/kernel/policy/executor.ts @@ -6,6 +6,7 @@ import type { Logger } from "../../contracts/logger.ts"; import type { AggregatesRuntime } from "../aggregate/runtime.ts"; import type { CommandPipeline } from "../command/pipeline.ts"; import { createReactionCommands } from "../command/reaction-commands.ts"; +import type { PersonalCategories } from "../personal-data/categories.ts"; import { openedUnder } from "../personal-data/event-store.ts"; import type { ForgetIntentions } from "../personal-data/intention.ts"; import { actedBy, causedBy } from "../shared/causation.ts"; @@ -49,6 +50,7 @@ export interface CreatePolicyExecutorArgs { readonly aggregates: AggregatesRuntime; readonly pipeline: CommandPipeline; readonly config: ResolvedConfig; + readonly personalCategories: PersonalCategories; readonly ids: IdGenerator; readonly clock: Clock; readonly logger: Logger; @@ -60,6 +62,7 @@ export interface CreatePolicyExecutorFunction { } export const createPolicyExecutor: CreatePolicyExecutorFunction = ({ + personalCategories, aggregates, pipeline, config, @@ -86,6 +89,8 @@ export const createPolicyExecutor: CreatePolicyExecutorFunction = ({ logger, ids, clock, + personalCategories, + batchSize: config.runtime.dispatcher.batchSize, intentions, maxChainDepth: config.forAggregate(policy.aggregate).policies.maxChainDepth, }); diff --git a/packages/core/src/kernel/process/failures.test.ts b/packages/core/src/kernel/process/failures.test.ts index d443de07..e3232fe2 100644 --- a/packages/core/src/kernel/process/failures.test.ts +++ b/packages/core/src/kernel/process/failures.test.ts @@ -1,5 +1,6 @@ import { afterEach, describe, expect, it } from "vitest"; import type { NewDeadLetter } from "../../adapter/storage/dead-letter-store.ts"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { pendingEvent } from "../../adapter/testing/fixtures.ts"; import { createFixedClock } from "../../contracts/clock.ts"; import { ValidationError } from "../../contracts/errors.ts"; @@ -199,7 +200,7 @@ describe("letterOf", () => { it("seals the message and the stack under the key the subject's person holds", async () => { const { failures, storage } = await setup(); - await storage.personalKeys.add("c-1", await newStoredKey()); + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, await newStoredKey()); const error = new Error("cannot mail ana@example.com"); const letter = await failures.letterOf( orderPayment, @@ -212,7 +213,7 @@ describe("letterOf", () => { expect(isSealed(letter.errorMessage)).toBe(true); expect(isSealed(letter.errorStack)).toBe(true); expect(JSON.stringify(letter)).not.toContain("ana@example.com"); - const key = await storage.personalKeys.get("c-1"); + const key = await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY); expect(key).not.toBeNull(); expect(letter.errorMessage).toMatchObject({ $personal: { person: "c-1", key: key?.fingerprint }, @@ -222,7 +223,7 @@ describe("letterOf", () => { it("seals the text unreadable when the person no longer holds the key the subject was read under", async () => { const { failures, storage } = await setup(); const read = await newStoredKey(); - await storage.personalKeys.add("c-1", await newStoredKey()); + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, await newStoredKey()); const letter = await failures.letterOf( orderPayment, { ...subject, person: "c-1", read: [{ person: "c-1", fingerprint: read.fingerprint }] }, @@ -232,7 +233,9 @@ describe("letterOf", () => { ); expect(isSealed(letter.errorMessage)).toBe(true); expect(letter.errorMessage).not.toMatchObject({ - $personal: { key: (await storage.personalKeys.get("c-1"))?.fingerprint }, + $personal: { + key: (await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY))?.fingerprint, + }, }); expect(letter.errorMessage).not.toMatchObject({ $personal: { key: read.fingerprint } }); }); @@ -247,7 +250,7 @@ describe("letterOf", () => { "terminal", ); expect(isSealed(letter.errorMessage)).toBe(true); - expect(await storage.personalKeys.get("c-1")).toBeNull(); + expect(await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY)).toBeNull(); }); }); @@ -286,7 +289,11 @@ describe("filed", () => { describe("subjectOf an event on a loaded instance", () => { const onboarding: ProcessRuntime = { ...orderPayment, - personal: { person: "customerId", fields: ["email"] }, + personal: { + person: "customerId", + fields: ["email"], + categories: { email: DEFAULT_PERSONAL_CATEGORY }, + }, }; const held = { person: "c-2", fingerprint: "f-2" }; const loaded = instance({ state: { customerId: "c-2", email: "a@b.c" }, persons: [held] }); diff --git a/packages/core/src/kernel/process/handlers.ts b/packages/core/src/kernel/process/handlers.ts index e0e6e6f8..aa6b7cb4 100644 --- a/packages/core/src/kernel/process/handlers.ts +++ b/packages/core/src/kernel/process/handlers.ts @@ -9,6 +9,7 @@ import type { CausationContext } from "../../contracts/metadata.ts"; import type { AggregatesRuntime } from "../aggregate/runtime.ts"; import type { CommandPipeline } from "../command/pipeline.ts"; import { createReactionCommands, type ReactionCommands } from "../command/reaction-commands.ts"; +import type { PersonalCategories } from "../personal-data/categories.ts"; import { openedUnder } from "../personal-data/event-store.ts"; import type { ForgetIntentions } from "../personal-data/intention.ts"; import { checkStatePerson } from "../personal-data/process-state.ts"; @@ -87,6 +88,7 @@ export interface CreateProcessHandlersArgs { readonly aggregates: AggregatesRuntime; readonly pipeline: CommandPipeline; readonly config: ResolvedConfig; + readonly personalCategories: PersonalCategories; readonly ids: IdGenerator; readonly clock: Clock; readonly logger: Logger; @@ -98,6 +100,7 @@ export interface CreateProcessHandlersFunction { } export const createProcessHandlers: CreateProcessHandlersFunction = ({ + personalCategories, aggregates, pipeline, config, @@ -121,6 +124,8 @@ export const createProcessHandlers: CreateProcessHandlersFunction = ({ logger, ids, clock, + personalCategories, + batchSize: config.runtime.dispatcher.batchSize, intentions, maxChainDepth: config.forAggregate(process.aggregate).policies.maxChainDepth, }); diff --git a/packages/core/src/kernel/process/instances.test.ts b/packages/core/src/kernel/process/instances.test.ts index e49e5a62..c34ff466 100644 --- a/packages/core/src/kernel/process/instances.test.ts +++ b/packages/core/src/kernel/process/instances.test.ts @@ -1,12 +1,13 @@ import { describe, expect, it } from "vitest"; import { ForgottenSinceLoadError } from "../../adapter/storage/append-checks.ts"; +import { DEFAULT_PERSONAL_CATEGORY } from "../../adapter/storage/personal-key-store.ts"; import { pendingEvent } from "../../adapter/testing/fixtures.ts"; import { createFixedClock } from "../../contracts/clock.ts"; import { createSequentialIdGenerator } from "../../contracts/ids.ts"; import { memory } from "../../memory/index.ts"; import { buildAggregates } from "../aggregate/build-aggregates.ts"; import { withPersonalData } from "../personal-data/event-store.ts"; -import { keysOver } from "../test-support.ts"; +import { keysOver, removeKeys } from "../test-support.ts"; import { createUnitOfWork } from "../unit-of-work/unit-of-work.ts"; import type { ProcessRuntime } from "./build-processes.ts"; import { createProcessInstances } from "./instances.ts"; @@ -79,7 +80,11 @@ describe("createProcessInstances", () => { const storage = await memory().createStorage(); const onboarding: ProcessRuntime = { ...process, - personal: { person: "customerId", fields: ["email"] }, + personal: { + person: "customerId", + fields: ["email"], + categories: { email: DEFAULT_PERSONAL_CATEGORY }, + }, }; const instances = createProcessInstances({ eventStore: withPersonalData({ @@ -99,11 +104,12 @@ describe("createProcessInstances", () => { expect(fresh.persons).toEqual([]); await instances.append(onboarding, "o-1", fresh, [handled]); const loaded = await instances.load(onboarding, "o-1"); - const fingerprint = (await storage.personalKeys.get("c-1"))?.fingerprint ?? null; + const fingerprint = + (await storage.personalKeys.get("c-1", DEFAULT_PERSONAL_CATEGORY))?.fingerprint ?? null; expect(loaded.state).toEqual(state); expect(loaded.persons).toEqual([{ person: "c-1", fingerprint }]); - await storage.personalKeys.remove("c-1"); + await removeKeys(storage.personalKeys, "c-1"); await expect( instances.append(onboarding, "o-1", loaded, [lifecycleEntries.resumed(context)]), ).rejects.toThrow(ForgottenSinceLoadError); diff --git a/packages/core/src/kernel/process/instances.ts b/packages/core/src/kernel/process/instances.ts index d6ac9638..dd669082 100644 --- a/packages/core/src/kernel/process/instances.ts +++ b/packages/core/src/kernel/process/instances.ts @@ -68,14 +68,14 @@ export const createProcessInstances: CreateProcessInstancesFunction = ({ const load = async (process: ProcessRuntime, instanceId: string): Promise => { const stream = { aggregateType: processAggregateType(process.name), aggregateId: instanceId }; - const { events, fingerprints = new Map() } = await eventStore.load(stream); + const { events, opened = [] } = await eventStore.load(stream); const instance: Viewed = { ...foldProcess({ initialState: process.initialState, deadlineFields: process.deadlineFields, events, }), - persons: [...fingerprints].map(([person, fingerprint]) => ({ person, fingerprint })), + persons: opened, [VIEW]: view, }; return instance; diff --git a/packages/core/src/kernel/process/runner.ts b/packages/core/src/kernel/process/runner.ts index 93b307a6..b3e2bde1 100644 --- a/packages/core/src/kernel/process/runner.ts +++ b/packages/core/src/kernel/process/runner.ts @@ -6,6 +6,7 @@ import type { AggregatesRuntime } from "../aggregate/runtime.ts"; import type { CommandPipeline } from "../command/pipeline.ts"; import type { Subscriber } from "../dispatch/dispatcher.ts"; import type { AppStorage } from "../events/app-event-store.ts"; +import type { PersonalCategories } from "../personal-data/categories.ts"; import type { ForgetIntentions } from "../personal-data/intention.ts"; import { deliverInOrder } from "../shared/in-order.ts"; import type { PendingRetries } from "../shared/pending-retries.ts"; @@ -30,6 +31,7 @@ export interface CreateProcessRunnerArgs { readonly pipeline: CommandPipeline; readonly storage: AppStorage; readonly config: ResolvedConfig; + readonly personalCategories: PersonalCategories; readonly ids: IdGenerator; readonly clock: Clock; readonly pendingRetries: PendingRetries; @@ -46,6 +48,7 @@ export interface CreateProcessRunnerFunction { * overtakes it. */ export const createProcessRunner: CreateProcessRunnerFunction = ({ + personalCategories, processes, aggregates, pipeline, @@ -69,6 +72,7 @@ export const createProcessRunner: CreateProcessRunnerFunction = ({ aggregates, pipeline, config, + personalCategories, ids, clock, logger, diff --git a/packages/core/src/kernel/reactive-harness.ts b/packages/core/src/kernel/reactive-harness.ts index fdfec034..a406308f 100644 --- a/packages/core/src/kernel/reactive-harness.ts +++ b/packages/core/src/kernel/reactive-harness.ts @@ -14,6 +14,7 @@ import { createRestrictedRetries } from "./dead-letters/restricted.ts"; import { createDeadLetterRetry, type DeadLetterRetry } from "./dead-letters/retry.ts"; import { createDispatcher, type Dispatcher } from "./dispatch/dispatcher.ts"; import type { AppStorage } from "./events/app-event-store.ts"; +import { buildPersonalCategories } from "./personal-data/categories.ts"; import { createForgetIntentions } from "./personal-data/intention.ts"; import { buildPolicies, type PoliciesRuntime } from "./policy/build-policies.ts"; import { createDelayedPolicies } from "./policy/delayed.ts"; @@ -105,11 +106,16 @@ export const createReactiveHarness: CreateReactiveHarnessFunction = async ({ logger, }); const policies = buildPolicies({ registry, aggregates }); + const personalCategories = buildPersonalCategories({ + module: registry.personalData, + retention: config.runtime.personalData.retention, + }); const intentions = createForgetIntentions({ ledger: null, ids, clock, logger }); const policyExecutor = createPolicyExecutor({ aggregates, pipeline, config, + personalCategories, ids, clock, logger, @@ -121,6 +127,7 @@ export const createReactiveHarness: CreateReactiveHarnessFunction = async ({ pipeline, storage, config, + personalCategories, ids, clock, pendingRetries: ignoredRetries, diff --git a/packages/core/src/kernel/status.ts b/packages/core/src/kernel/status.ts index 48eb865e..12d727be 100644 --- a/packages/core/src/kernel/status.ts +++ b/packages/core/src/kernel/status.ts @@ -62,7 +62,9 @@ export interface AppStatus { */ readonly keyRotation: KeyRotationStatus | null; /** - * `null` without `runtime.personalData.retention`. + * `null` when no schema marks personal data: every retention set, of the app and of each + * declared category, and the keys kept past a forgetting that no legal obligation keeps any + * more. */ readonly retention: RetentionStatus | null; /** diff --git a/packages/core/src/kernel/test-support.ts b/packages/core/src/kernel/test-support.ts index 9273b0b0..32ffca8a 100644 --- a/packages/core/src/kernel/test-support.ts +++ b/packages/core/src/kernel/test-support.ts @@ -459,6 +459,20 @@ export const withPersonalKeys: WithPersonalKeysFunction = (storage, clock = crea }; }; +export interface RemoveKeysFunction { + (keys: PersonalKeyStore, person: string): Promise; +} + +/** + * Removes every key the person holds, as forgetting the whole person on another instance would, + * and reports whether they held any. + */ +export const removeKeys: RemoveKeysFunction = async (keys, person) => { + const held = await keys.list(person); + for (const { key } of held) await keys.remove(person, key.fingerprint); + return held.length > 0; +}; + export interface NewStoredKeyFunction { (): Promise; } @@ -485,6 +499,7 @@ export interface OpenedKeyFunction { export const openedKey: OpenedKeyFunction = async ({ key, fingerprint }) => ({ fingerprint, cryptoKey: await importRawKey({ raw: fromBase64(key), extractable: false }), + retained: false, }); export interface WithoutLedgerArgs { diff --git a/packages/core/src/kernel/unit-of-work/unit-of-work.test.ts b/packages/core/src/kernel/unit-of-work/unit-of-work.test.ts index d5cb63d8..a036afb6 100644 --- a/packages/core/src/kernel/unit-of-work/unit-of-work.test.ts +++ b/packages/core/src/kernel/unit-of-work/unit-of-work.test.ts @@ -333,7 +333,7 @@ describe("createUnitOfWork", () => { it("runs the steps before the commit's transaction, in order, on the store, and writes nothing when one throws", async () => { const storage = await memory().createStorage(); - await storage.personalKeys.add("c-1", { key: "k", fingerprint: "f-1", kid: null }); + await storage.personalKeys.add("c-1", "default", { key: "k", fingerprint: "f-1", kid: null }); const transact = storage.transact.bind(storage); const ran: string[] = []; storage.transact = (work) => { @@ -342,7 +342,7 @@ describe("createUnitOfWork", () => { }; const unit = createUnitOfWork({ storage }); unit.beforeCommit(async ({ personalKeys }) => { - ran.push(`first ${(await personalKeys.get("c-1"))?.fingerprint}`); + ran.push(`first ${(await personalKeys.get("c-1", "default"))?.fingerprint}`); }); unit.beforeCommit(async () => { ran.push("second"); @@ -589,12 +589,25 @@ describe("createUnitOfWork", () => { }), ).rejects.toBe(failure); } - expect(readUnder({ person: "c-1", error: failure })).toEqual(["f-1", "f-2"]); + const fingerprints = (args: Parameters[0]) => + readUnder(args).map(({ fingerprint }) => fingerprint); + expect(fingerprints({ person: "c-1", error: failure })).toEqual(["f-1", "f-2"]); expect(readUnder({ person: "c-2", error: failure })).toEqual([]); expect( - readUnder({ person: "c-1", error: failure, also: [{ person: "c-1", fingerprint: "f-3" }] }), + fingerprints({ + person: "c-1", + error: failure, + also: [{ person: "c-1", fingerprint: "f-3", retained: true }], + }), ).toEqual(["f-1", "f-2", "f-3"]); - expect(readUnder({ person: "c-1", error: "a string", also: [first] })).toEqual(["f-1"]); + expect( + readUnder({ + person: "c-1", + error: failure, + also: [{ person: "c-1", fingerprint: "f-3", retained: true }], + }).at(-1), + ).toEqual({ person: "c-1", fingerprint: "f-3", retained: true }); + expect(fingerprints({ person: "c-1", error: "a string", also: [first] })).toEqual(["f-1"]); const wrapped = await commitAttempt({ storage, concurrencyRetries: 0, @@ -610,7 +623,7 @@ describe("createUnitOfWork", () => { value: "mailer down", cause: "mailer down", }); - expect(readUnder({ person: "c-1", error: wrapped })).toEqual(["f-1"]); + expect(fingerprints({ person: "c-1", error: wrapped })).toEqual(["f-1"]); expect(errorIdentity(wrapped)).toEqual({ errorName: "string" }); expect(errorDetails(wrapped)).toEqual({ message: "mailer down" }); expect(asError(failure)).toBe(failure); diff --git a/packages/core/src/memory/dead-letter-store.ts b/packages/core/src/memory/dead-letter-store.ts index 1e5963da..622d1ff9 100644 --- a/packages/core/src/memory/dead-letter-store.ts +++ b/packages/core/src/memory/dead-letter-store.ts @@ -102,8 +102,8 @@ export const createKeptDeadLetterStore: CreateKeptDeadLetterStoreFunction = () = select({}) .filter((letter) => letter.person === person || namesPerson(letter, person)) .map(copy), - forget: async (person) => { - const named = select({}).filter((letter) => namesPerson(letter, person)); + forget: async (person, keep) => { + const named = select({}).filter((letter) => namesPerson(letter, person, keep)); for (const { id } of named) letters.delete(id); return named.map(copy); }, diff --git a/packages/core/src/memory/idempotency-store.ts b/packages/core/src/memory/idempotency-store.ts index 470e4042..b237e17b 100644 --- a/packages/core/src/memory/idempotency-store.ts +++ b/packages/core/src/memory/idempotency-store.ts @@ -1,5 +1,5 @@ import type { IdempotencyRecord, IdempotencyStore } from "../adapter/storage/idempotency-store.ts"; -import type { KeyedPerson } from "../adapter/storage/personal-key-store.ts"; +import { type KeyedPerson, namesPerson } from "../adapter/storage/personal-key-store.ts"; import { IdempotencyKeyTakenError } from "../contracts/errors.ts"; import { createStoreEntries, type WithEntries } from "./entries.ts"; import { jsonCopy } from "./json-copy.ts"; @@ -48,9 +48,9 @@ export const createKeptIdempotencyStore: CreateKeptIdempotencyStoreFunction = ({ } return purged; }, - forget: async (person) => { + forget: async (person, keep) => { for (const [key, record] of records) { - if (record.persons.some((held) => held.person === person)) records.delete(key); + if (namesPerson(record, person, keep)) records.delete(key); } }, }; diff --git a/packages/core/src/memory/index.ts b/packages/core/src/memory/index.ts index 18719dfc..c80b29c3 100644 --- a/packages/core/src/memory/index.ts +++ b/packages/core/src/memory/index.ts @@ -131,15 +131,18 @@ export const memory: MemoryFunction = (options = {}) => { const inboxLedger = createKeptInboxLedger(); const deadLetterStore = createKeptDeadLetterStore(); const scheduler = createKeptScheduler(); - const personalKeys = createKeptPersonalKeyStore(); const persons = createKeptPersonStore(); + const personalKeys = createKeptPersonalKeyStore({ + restricted: persons.restricted, + lastActiveAt: persons.lastActiveAt, + }); const idempotencyStore = createKeptIdempotencyStore({ holds: personalKeys.holds }); const snapshotStore = createMemorySnapshotStore({ holds: personalKeys.holds }); const subscriberFailures = createMemorySubscriberFailureStore(); const stores = { eventStore: createMemoryEventStore({ onAppend: notifier.notify, - checks: { fingerprintOf: personalKeys.fingerprintOf, restricted: persons.restricted }, + checks: { keyOf: personalKeys.keyOf, restricted: persons.restricted }, }), inboxLedger: inboxLedger.store, deadLetterStore: deadLetterStore.store, diff --git a/packages/core/src/memory/memory.test.ts b/packages/core/src/memory/memory.test.ts index 9175c23f..fa1ae03e 100644 --- a/packages/core/src/memory/memory.test.ts +++ b/packages/core/src/memory/memory.test.ts @@ -39,7 +39,7 @@ describe("memory adapter", () => { idempotencyStoreContract({ create: storage }); snapshotStoreContract({ create: storage }); storageTransactionContract({ create: storage }); - personalKeyStoreContract({ create: async () => (await storage()).personalKeys }); + personalKeyStoreContract({ create: async () => storage() }); personStoreContract({ create: async () => (await storage()).persons }); inspectionContract({ create: async () => memory() }); subscriberFailureStoreContract({ create: async () => (await storage()).subscriberFailures }); diff --git a/packages/core/src/memory/person-store.ts b/packages/core/src/memory/person-store.ts index d925554d..c2c79b24 100644 --- a/packages/core/src/memory/person-store.ts +++ b/packages/core/src/memory/person-store.ts @@ -9,15 +9,18 @@ export interface KeptPersonStore extends WithEntries { * store checks it in the same step as it writes. */ restricted(person: string): boolean; + /** + * The person's last activity recorded since their last forgetting as a whole, answered at once; + * `null` without one. + */ + lastActiveAt(person: string): string | null; } export interface CreateKeptPersonStoreFunction { (): KeptPersonStore; } -export interface KeptPerson extends StoredPerson { - readonly position: number; -} +export type KeptPerson = StoredPerson; export interface BlankPersonFunction { (person: string): KeptPerson; @@ -33,7 +36,7 @@ export const blankPerson: BlankPersonFunction = (person) => ({ position: 0, }); -const stored = ({ position: _position, ...person }: KeptPerson): StoredPerson => person; +const stored = (person: KeptPerson): StoredPerson => ({ ...person }); // As SQL orders text, by code unit rather than by locale. const compare = (a: string, b: string): number => (a < b ? -1 : a > b ? 1 : 0); @@ -49,6 +52,10 @@ export const createKeptPersonStore: CreateKeptPersonStoreFunction = () => { const existing = (person: string): KeptPerson => persons.get(person) ?? blankPerson(person); return { restricted: (person) => (persons.get(person)?.restrictedSince ?? null) !== null, + lastActiveAt: (person) => { + const kept = persons.get(person); + return kept === undefined || kept.forgottenAt !== null ? null : kept.lastActiveAt; + }, store: { get: async (person) => { const kept = persons.get(person); diff --git a/packages/core/src/memory/personal-key-store.ts b/packages/core/src/memory/personal-key-store.ts index f40046d2..aea3612f 100644 --- a/packages/core/src/memory/personal-key-store.ts +++ b/packages/core/src/memory/personal-key-store.ts @@ -1,101 +1,251 @@ +import type { KeyState } from "../adapter/storage/append-checks.ts"; import type { Hold } from "../adapter/storage/hold.ts"; import { isReleasedBy } from "../adapter/storage/hold.ts"; -import type { - KeyedPerson, - PersonalKeyStore, - StoredPersonalKey, +import { + type DueKey, + type HeldPersonalKey, + type KeyedPerson, + keyedPersonId, + type PersonalKeyStore, + type StoredPersonalKey, } from "../adapter/storage/personal-key-store.ts"; import { createStoreEntries, type WithEntries } from "./entries.ts"; import { earliestHoldEnd } from "./holds.ts"; export interface KeptPersonalKeyStore extends WithEntries { /** - * Whether the person holds the key of that fingerprint, always for a `null` one, answered - * at once: the memory snapshot store checks it in the same step as it writes, so no forget runs - * between the two. + * Whether the person holds the key of that fingerprint in force, always for a `null` one, + * answered at once: the memory snapshot store checks it in the same step as it writes, so no + * forget runs between the two. */ holds(held: KeyedPerson): boolean; /** - * The fingerprint of the person's key, `null` when they hold none, answered at once for the - * same reason. + * The key of that fingerprint as the person holds it, answered at once for the same reason. */ - fingerprintOf(person: string): string | null; + keyOf(person: string, fingerprint: string): KeyState; +} + +export interface CreateKeptPersonalKeyStoreArgs { + /** + * Whether the processing of the person's data is restricted: `listDue` leaves them out. + */ + readonly restricted: (person: string) => boolean; + /** + * The person's last activity recorded since their last forgetting as a whole, which a new key + * starts from; `null` without one. + */ + readonly lastActiveAt: (person: string) => string | null; } export interface CreateKeptPersonalKeyStoreFunction { - (): KeptPersonalKeyStore; + (args: CreateKeptPersonalKeyStoreArgs): KeptPersonalKeyStore; } interface KeptPersonalKey extends StoredPersonalKey { + readonly person: string; + readonly category: string; readonly hold: Hold | null; + readonly lastActiveAt: string | null; + readonly retainedSince: string | null; + readonly dueHold: Hold | null; } -const stored = ({ hold: _hold, ...key }: KeptPersonalKey): StoredPersonalKey => key; +const stored = ({ key, fingerprint, kid }: KeptPersonalKey): StoredPersonalKey => ({ + key, + fingerprint, + kid, +}); + +const held = (kept: KeptPersonalKey): HeldPersonalKey => ({ + category: kept.category, + key: stored(kept), + retainedSince: kept.retainedSince, + lastActiveAt: kept.lastActiveAt, +}); const isHeldBy = ( - held: KeptPersonalKey | undefined, + kept: KeptPersonalKey | undefined, from: StoredPersonalKey, -): held is KeptPersonalKey => held?.key === from.key && held.fingerprint === from.fingerprint; +): kept is KeptPersonalKey => kept?.key === from.key && kept.fingerprint === from.fingerprint; -export const createKeptPersonalKeyStore: CreateKeptPersonalKeyStoreFunction = () => { +// Entries are named as `keyedPersonId` names a person under a key. +const idOf = (person: string, fingerprint: string): string => + keyedPersonId({ person, fingerprint }); + +const dueKeyOf = (kept: KeptPersonalKey): DueKey => ({ + person: kept.person, + category: kept.category, + fingerprint: kept.fingerprint, + lastActiveAt: kept.lastActiveAt ?? "", + retainedSince: kept.retainedSince, + hold: kept.dueHold, +}); + +// Persons, then fingerprints, are unique together, so no two compare equal. +const byPerson = (a: KeptPersonalKey, b: KeptPersonalKey): number => + a.person === b.person ? (a.fingerprint < b.fingerprint ? -1 : 1) : a.person < b.person ? -1 : 1; + +export const createKeptPersonalKeyStore: CreateKeptPersonalKeyStoreFunction = ({ + restricted, + lastActiveAt: lastActiveOf, +}) => { const keys = new Map(); + const of = (person: string): KeptPersonalKey[] => + [...keys.values()].filter((kept) => kept.person === person); + const inForce = (person: string, category: string): KeptPersonalKey | undefined => + of(person).find((kept) => kept.category === category && kept.retainedSince === null); + const update = (kept: KeptPersonalKey, change: Partial): void => { + keys.set(idOf(kept.person, kept.fingerprint), { ...kept, ...change }); + }; + const due = (category: string, releasedBy: Date | undefined): KeptPersonalKey[] => + [...keys.values()].filter( + (kept) => + kept.category === category && + kept.lastActiveAt !== null && + !restricted(kept.person) && + isReleasedBy(kept.dueHold, releasedBy), + ); + const retainedOutside = (except: readonly string[]): KeptPersonalKey[] => + [...keys.values()].filter( + (kept) => + kept.retainedSince !== null && !except.includes(kept.category) && !restricted(kept.person), + ); return { - holds: ({ person, fingerprint }) => { - const stored = keys.get(person); - return fingerprint === null || stored?.fingerprint === fingerprint; + holds: ({ person, fingerprint }) => + fingerprint === null || keys.get(idOf(person, fingerprint))?.retainedSince === null, + keyOf: (person, fingerprint) => { + const kept = keys.get(idOf(person, fingerprint)); + if (kept === undefined) return "gone"; + return kept.retainedSince === null ? "held" : "retained"; }, - fingerprintOf: (person) => keys.get(person)?.fingerprint ?? null, store: { - get: async (person) => { - const kept = keys.get(person); + get: async (person, category) => { + const kept = inForce(person, category); return kept === undefined ? null : stored(kept); }, - add: async (person, key) => { - const existing = keys.get(person); + find: async (person, fingerprint) => { + const kept = keys.get(idOf(person, fingerprint)); + return kept === undefined ? null : held(kept); + }, + list: async (person) => of(person).sort(byPerson).map(held), + add: async (person, category, key) => { + const existing = inForce(person, category); if (existing !== undefined) return stored(existing); - const kept = { key: key.key, fingerprint: key.fingerprint, kid: key.kid, hold: null }; - keys.set(person, kept); + const kept: KeptPersonalKey = { + person, + category, + key: key.key, + fingerprint: key.fingerprint, + kid: key.kid, + hold: null, + lastActiveAt: lastActiveOf(person), + retainedSince: null, + dueHold: null, + }; + keys.set(idOf(person, key.fingerprint), kept); return stored(kept); }, - remove: async (person) => keys.delete(person), - listWrappedUnder: async ({ kids, releasedBy, limit }) => { - return ( - [...keys.entries()] - .filter(([, key]) => kids.includes(key.kid)) - .filter(([, { hold }]) => isReleasedBy(hold, releasedBy)) - // Persons are unique, so no two compare equal. - .sort(([a], [b]) => (a < b ? -1 : 1)) - .slice(0, limit) - .map(([person, key]) => ({ person, key: stored(key), hold: key.hold })) - ); + remove: async (person, fingerprint) => keys.delete(idOf(person, fingerprint)), + retain: async (person, fingerprint, { since, lastActiveAt }) => { + const kept = keys.get(idOf(person, fingerprint)); + if (kept === undefined || kept.retainedSince !== null) return false; + update(kept, { retainedSince: since, lastActiveAt, dueHold: null }); + return true; + }, + touch: async (person, lastActiveAt) => { + for (const kept of of(person)) { + if (kept.retainedSince !== null) continue; + if (kept.lastActiveAt !== null && kept.lastActiveAt >= lastActiveAt) continue; + update(kept, { lastActiveAt, dueHold: null }); + } + }, + listDue: async ({ category, activeBefore, releasedBy, limit }) => { + const before = activeBefore?.toISOString(); + return due(category, releasedBy) + .filter((kept) => before === undefined || (kept.lastActiveAt ?? "") <= before) + .sort((a, b) => + a.lastActiveAt === b.lastActiveAt + ? byPerson(a, b) + : (a.lastActiveAt ?? "") < (b.lastActiveAt ?? "") + ? -1 + : 1, + ) + .slice(0, limit) + .map(dueKeyOf); + }, + holdDue: async (person, fingerprint, lastActiveAt, hold) => { + const kept = keys.get(idOf(person, fingerprint)); + if (kept?.lastActiveAt === lastActiveAt) update(kept, { dueHold: hold }); }, + nextDueHoldEnd: async ({ category, after }) => + earliestHoldEnd( + due(category, undefined).map(({ dueHold }) => dueHold), + after, + ), + listRetained: async ({ except, releasedBy, limit }) => + retainedOutside(except) + .filter((kept) => isReleasedBy(kept.dueHold, releasedBy)) + .sort((a, b) => + a.retainedSince === b.retainedSince + ? byPerson(a, b) + : (a.retainedSince ?? "") < (b.retainedSince ?? "") + ? -1 + : 1, + ) + .slice(0, limit) + .map(dueKeyOf), + nextRetainedHoldEnd: async ({ except, after }) => + earliestHoldEnd( + retainedOutside(except).map(({ dueHold }) => dueHold), + after, + ), + listWrappedUnder: async ({ kids, releasedBy, limit }) => + [...keys.values()] + .filter((kept) => kids.includes(kept.kid)) + .filter(({ hold }) => isReleasedBy(hold, releasedBy)) + .sort(byPerson) + .slice(0, limit) + .map((kept) => ({ + person: kept.person, + category: kept.category, + key: stored(kept), + hold: kept.hold, + })), rewrap: async (person, from, to) => { - const held = keys.get(person); - if (!isHeldBy(held, from)) return false; - keys.set(person, { ...held, key: to.key, kid: to.kid, hold: null }); + const kept = keys.get(idOf(person, from.fingerprint)); + if (!isHeldBy(kept, from)) return false; + update(kept, { key: to.key, kid: to.kid, hold: null }); return true; }, hold: async (person, from, hold) => { - const held = keys.get(person); - if (isHeldBy(held, from)) keys.set(person, { ...held, hold }); + const kept = keys.get(idOf(person, from.fingerprint)); + if (isHeldBy(kept, from)) update(kept, { hold }); }, nextHoldEnd: async ({ kids, after }) => earliestHoldEnd( - [...keys.values()].filter((key) => kids.includes(key.kid)).map(({ hold }) => hold), + [...keys.values()].filter((kept) => kids.includes(kept.kid)).map(({ hold }) => hold), after, ), countWrappedUnder: async ({ kids }) => [...keys.values()].filter((key) => kids.includes(key.kid)).length, releaseHolds: async () => { - let released = 0; - for (const [person, kept] of keys) { - if (kept.hold === null) continue; - keys.set(person, { ...kept, hold: null }); - released += 1; - } - return released; + const held = [...keys.values()].filter(({ hold }) => hold !== null); + for (const kept of held) update(kept, { hold: null }); + return held.length; + }, + countDue: async ({ category, activeBefore }) => { + const before = activeBefore?.toISOString(); + return due(category, undefined).filter( + (kept) => before === undefined || (kept.lastActiveAt ?? "") <= before, + ).length; + }, + countRetained: async ({ except }) => retainedOutside(except).length, + releaseDueHolds: async () => { + const held = [...keys.values()].filter(({ dueHold }) => dueHold !== null); + for (const kept of held) update(kept, { dueHold: null }); + return held.length; }, }, - entries: createStoreEntries(keys, (person) => person), + entries: createStoreEntries(keys, (id: string) => id), }; }; diff --git a/packages/core/src/memory/scheduler.ts b/packages/core/src/memory/scheduler.ts index d3cb63db..7efd697c 100644 --- a/packages/core/src/memory/scheduler.ts +++ b/packages/core/src/memory/scheduler.ts @@ -183,8 +183,10 @@ export const createKeptScheduler: CreateKeptSchedulerFunction = () => { .slice(offset, limit === undefined ? undefined : offset + limit) .map(toScheduled), listByPerson: async (person) => naming(person).map(toScheduled), - forget: async (person) => { - const named = naming(person); + forget: async (person, keep) => { + const named = [...entries.values()] + .filter((entry) => namesPerson(entry, person, keep)) + .sort(byExecuteAt); for (const { dedupeKey } of named) entries.delete(dedupeKey); return named.map(toScheduled); }, diff --git a/packages/core/src/memory/snapshot-store.ts b/packages/core/src/memory/snapshot-store.ts index 0299cecf..0c10f20e 100644 --- a/packages/core/src/memory/snapshot-store.ts +++ b/packages/core/src/memory/snapshot-store.ts @@ -1,4 +1,4 @@ -import type { KeyedPerson } from "../adapter/storage/personal-key-store.ts"; +import { type KeyedPerson, namesPerson } from "../adapter/storage/personal-key-store.ts"; import type { Snapshot, SnapshotStore } from "../adapter/storage/snapshot-store.ts"; import { streamId } from "../contracts/event.ts"; @@ -34,9 +34,9 @@ export const createMemorySnapshotStore: CreateMemorySnapshotStoreFunction = ({ h remove: async (stream) => { snapshots.delete(streamId(stream)); }, - forget: async (person) => { + forget: async (person, keep) => { for (const [key, snapshot] of snapshots) { - if (snapshot.persons.some((held) => held.person === person)) snapshots.delete(key); + if (namesPerson(snapshot, person, keep)) snapshots.delete(key); } }, }; diff --git a/packages/core/src/memory/transaction.test.ts b/packages/core/src/memory/transaction.test.ts index 15f84b1c..33ceb532 100644 --- a/packages/core/src/memory/transaction.test.ts +++ b/packages/core/src/memory/transaction.test.ts @@ -29,11 +29,11 @@ describe("createMemoryStorageTransaction", () => { it("checks at commit a key it gave a person after removing their old one", async () => { const storage = await memory().createStorage(); - await storage.personalKeys.add("c-1", testPersonalKey("one")); + await storage.personalKeys.add("c-1", "default", testPersonalKey("one")); await expect( storage.transact(async (tx) => { - await tx.personalKeys.remove("c-1"); - await tx.personalKeys.add("c-1", testPersonalKey("two")); + await tx.personalKeys.remove("c-1", "f-one"); + await tx.personalKeys.add("c-1", "default", testPersonalKey("two")); await tx.eventStore.append({ aggregateType: "order", aggregateId: "1", @@ -42,12 +42,50 @@ describe("createMemoryStorageTransaction", () => { persons: [testKeyedPerson("c-1", "one"), testKeyedPerson("c-1", "two")], }); // Another writer removes the key this transaction gave, before it commits. - await storage.personalKeys.remove("c-1"); + await storage.personalKeys.remove("c-1", "f-two"); }), ).rejects.toBeInstanceOf(ForgottenSinceLoadError); expect(await storage.eventStore.lastPosition()).toBe(0); }); + it("checks at commit a read made under a key in force that someone retained since", async () => { + const stores = await storage(); + await stores.personalKeys.add("c-1", "invoices", testPersonalKey("two")); + await expect( + stores.transact(async (tx) => { + await tx.eventStore.append({ + aggregateType: "order", + aggregateId: "1", + expectedVersion: 0, + events: [pendingEvent({ aggregateId: "1", version: 1 })], + persons: [testKeyedPerson("c-1", "two")], + }); + await stores.personalKeys.retain("c-1", "f-two", { + since: at(2).toISOString(), + lastActiveAt: at(1).toISOString(), + }); + }), + ).rejects.toBeInstanceOf(ForgottenSinceLoadError); + expect(await stores.eventStore.lastPosition()).toBe(0); + }); + + it("puts back the keys it retained or touched when it fails", async () => { + const stores = await storage(); + const day = (n: number) => at(n).toISOString(); + await stores.personalKeys.add("c-1", "default", testPersonalKey("one")); + await stores.personalKeys.add("c-1", "invoices", testPersonalKey("two")); + await stores.personalKeys.touch("c-1", day(1)); + const before = await stores.personalKeys.list("c-1"); + await expect( + stores.transact(async (tx) => { + await tx.personalKeys.retain("c-1", "f-two", { since: day(2), lastActiveAt: day(1) }); + await tx.personalKeys.touch("c-1", day(3)); + await refused(tx); + }), + ).rejects.toBeInstanceOf(ScheduledClaimLostError); + expect(await stores.personalKeys.list("c-1")).toEqual(before); + }); + it("puts back every entry it changed, newest first, when a later write fails", async () => { const stores = await storage(); await stores.scheduler.schedule(schedule("kept", 1)); @@ -90,6 +128,7 @@ describe("createMemoryStorageTransaction", () => { person: "c-1", restrictedSince: null, lastActiveAt: day(3), + position: 2, forgottenAt: null, hold: null, }); @@ -125,6 +164,7 @@ describe("createMemoryStorageTransaction", () => { person: "c-1", restrictedSince: null, lastActiveAt: day(1), + position: 1, forgottenAt: null, hold: null, }); diff --git a/packages/core/src/memory/transaction.ts b/packages/core/src/memory/transaction.ts index 36271fed..0761c28f 100644 --- a/packages/core/src/memory/transaction.ts +++ b/packages/core/src/memory/transaction.ts @@ -120,7 +120,8 @@ const undoAll = (undos: Undos): void => { }; // The keys are written at once, since `add` answers with the key in force, and put back when the -// transaction fails. Each key it removes goes in `removed`, as `keyedPersonId` names it. +// transaction fails. Each key it removes or retains goes in `removed`, as `keyedPersonId` names it: +// a read made while it was in force no longer holds. const trackedPersonalKeys = ( personalKeys: PersonalKeyStore, entries: StoreEntries, @@ -128,23 +129,56 @@ const trackedPersonalKeys = ( removed: Set, ): PersonalKeyStore => { const tracked = trackedBy(undos); + const id = (person: string, fingerprint: string): string => + keyedPersonId({ person, fingerprint }); return { - get: (person) => personalKeys.get(person), - add: (person, key) => tracked(entries, person, () => personalKeys.add(person, key)), - remove: async (person) => { - const held = await personalKeys.get(person); - if (held !== null) removed.add(keyedPersonId({ person, fingerprint: held.fingerprint })); - return tracked(entries, person, () => personalKeys.remove(person)); + get: (person, category) => personalKeys.get(person, category), + find: (person, fingerprint) => personalKeys.find(person, fingerprint), + list: (person) => personalKeys.list(person), + add: (person, category, key) => + tracked(entries, id(person, key.fingerprint), () => personalKeys.add(person, category, key)), + remove: async (person, fingerprint) => { + removed.add(id(person, fingerprint)); + return tracked(entries, id(person, fingerprint), () => + personalKeys.remove(person, fingerprint), + ); }, + retain: async (person, fingerprint, args) => { + removed.add(id(person, fingerprint)); + return tracked(entries, id(person, fingerprint), () => + personalKeys.retain(person, fingerprint, args), + ); + }, + // Several entries at once, each put back as `tracked` puts one back. + touch: async (person, lastActiveAt) => { + const ids = (await personalKeys.list(person)).map(({ key }) => id(person, key.fingerprint)); + const before = ids.map((at) => entries.read(at)); + await personalKeys.touch(person, lastActiveAt); + ids.forEach((at, index) => { + const after = entries.read(at); + undos.push(() => entries.putBack(at, before[index], after)); + }); + }, + listDue: (args) => personalKeys.listDue(args), + holdDue: (person, fingerprint, lastActiveAt, hold) => + tracked(entries, id(person, fingerprint), () => + personalKeys.holdDue(person, fingerprint, lastActiveAt, hold), + ), + nextDueHoldEnd: (args) => personalKeys.nextDueHoldEnd(args), + listRetained: (args) => personalKeys.listRetained(args), + nextRetainedHoldEnd: (args) => personalKeys.nextRetainedHoldEnd(args), listWrappedUnder: (args) => personalKeys.listWrappedUnder(args), rewrap: (person, from, to) => - tracked(entries, person, () => personalKeys.rewrap(person, from, to)), + tracked(entries, id(person, from.fingerprint), () => personalKeys.rewrap(person, from, to)), hold: (person, from, hold) => - tracked(entries, person, () => personalKeys.hold(person, from, hold)), + tracked(entries, id(person, from.fingerprint), () => personalKeys.hold(person, from, hold)), nextHoldEnd: (args) => personalKeys.nextHoldEnd(args), countWrappedUnder: (args) => personalKeys.countWrappedUnder(args), // Only after a restore, never inside a transaction. releaseHolds: () => personalKeys.releaseHolds(), + countDue: (args) => personalKeys.countDue(args), + countRetained: (args) => personalKeys.countRetained(args), + releaseDueHolds: () => personalKeys.releaseDueHolds(), }; }; @@ -195,28 +229,28 @@ const trackedForgets = ( return { scheduler: { ...stores.scheduler, - forget: async (person) => { - const named = (await scheduler.list()).filter((entry) => namesPerson(entry, person)); + forget: async (person, keep) => { + const named = (await scheduler.list()).filter((entry) => namesPerson(entry, person, keep)); for (const { dedupeKey } of named) { await tracked(entries.scheduler, dedupeKey, () => scheduler.cancel(dedupeKey)); } - const staged = withdrawScheduled(person); + const staged = withdrawScheduled(person, keep); const keys = new Set(staged.map(({ dedupeKey }) => dedupeKey)); return [...named.filter(({ dedupeKey }) => !keys.has(dedupeKey)), ...staged]; }, }, deadLetterStore: { ...stores.deadLetterStore, - forget: async (person) => { + forget: async (person, keep) => { const named = (await deadLetterStore.list()).filter((letter) => - namesPerson(letter, person), + namesPerson(letter, person, keep), ); for (const { id } of named) { await tracked(entries.deadLetterStore, id, () => deadLetterStore.remove(id)); } // A letter added under an id the store holds already changes nothing, so it stays as stored. const ids = new Set(named.map(({ id }) => id)); - return [...named, ...withdrawLetters(person).filter(({ id }) => !ids.has(id))]; + return [...named, ...withdrawLetters(person, keep).filter(({ id }) => !ids.has(id))]; }, }, }; @@ -263,7 +297,7 @@ const createUndoLog = ({ put: (args) => tracked(entries.idempotencyStore, args.key, () => idempotencyStore.put(args)), // Not put back: a rejection forgotten is decided again, which is all it costs. - forget: (person) => idempotencyStore.forget(person), + forget: (person, keep) => idempotencyStore.forget(person, keep), }, }, undo: () => undoAll(undos), diff --git a/packages/core/src/modules/bounda-events.ts b/packages/core/src/modules/bounda-events.ts index 71227588..d36f5744 100644 --- a/packages/core/src/modules/bounda-events.ts +++ b/packages/core/src/modules/bounda-events.ts @@ -1,4 +1,5 @@ import type { StoredEvent } from "../contracts/event.ts"; +import type { PersonalCategoryName } from "./personal-data.ts"; /** * The aggregate the runtime's own events belong to, one stream per subject: a read model @@ -35,8 +36,8 @@ export interface RemovedDeadLetter { /** * Why a person was forgotten: `requested` by `forget`, from a client or a policy or process, or - * by `bounda forget --from` for a bare id, or `retention`, past `runtime.personalData.retention` - * without activity. A forgetting committed again from `runtime.personalData.forgetLedger`, after + * by `bounda forget --from` for a bare id, or `retention`, past `runtime.personalData.retention`, + * or a category's own, without activity. A forgetting committed again from `runtime.personalData.forgetLedger`, after * a restore or a commit that never happened, keeps the reason it was asked with. */ export type PersonalDataForgottenReason = "requested" | "retention"; @@ -45,9 +46,31 @@ export interface PersonalDataForgottenPayload { readonly personId: string; readonly reason: PersonalDataForgottenReason; /** - * The fingerprints of the keys the forgetting destroyed, none when the person held none: a key - * the person holds whose fingerprint is not here was given to them after. Absent from a - * forgetting recorded before Bounda 0.4.0. + * Whether the person was forgotten as a whole, by `forget(person)`, the app's retention or a + * restore of either, with no key left in force; `false` when only some categories were, by + * `forget(person, { categories })` or a category's retention, or when a forgetting committed + * again kept a key given to the person after it. + */ + readonly whole: boolean; + /** + * The categories forgotten: their fields read as `null` from then on. As a whole, every + * category but those `retained`. + */ + readonly categories: readonly PersonalCategoryName[]; + /** + * The categories kept under a legal obligation instead, restricted until their retention ends, + * when another `PersonalDataForgotten` names them in `expired`. + */ + readonly retained: readonly PersonalCategoryName[]; + /** + * The categories kept at an earlier forgetting whose retention ended: what was kept of them is + * forgotten now. + */ + readonly expired: readonly PersonalCategoryName[]; + /** + * The fingerprints of the keys the forgetting covered, destroyed or retained, none when it + * covered none: a key in force the person holds whose fingerprint is not here was given to them + * after. Absent from a forgetting recorded before Bounda 0.4.0. */ readonly fingerprints?: readonly string[]; readonly cancelledCommands: readonly CancelledCommand[]; diff --git a/packages/core/src/modules/index.ts b/packages/core/src/modules/index.ts index a3750551..592d4556 100644 --- a/packages/core/src/modules/index.ts +++ b/packages/core/src/modules/index.ts @@ -1,3 +1,4 @@ +export { DEFAULT_PERSONAL_CATEGORY } from "../adapter/storage/personal-key-store.ts"; export type { AggregateMeta, HandlerState, @@ -75,10 +76,22 @@ export type { ForgetsPersonalData, HasPersonalFields, PersonalFunction, + PersonalOptions, PersonalSchema, PersonFunction, PersonSchema, } from "./personal.ts"; +export type { + DeclaresPersonalData, + ForgetOptions, + LegalBasis, + PersonalCategoriesOf, + PersonalCategoryName, + PersonalDataCategories, + PersonalDataCategory, + PersonalDataModule, +} from "./personal-data.ts"; +export { LEGAL_BASES } from "./personal-data.ts"; export type { PolicyHandlerArgs, PolicyModule, ReactionForgetFunction } from "./policy.ts"; export type { CreateArgs, diff --git a/packages/core/src/modules/personal-data.ts b/packages/core/src/modules/personal-data.ts new file mode 100644 index 00000000..199c9137 --- /dev/null +++ b/packages/core/src/modules/personal-data.ts @@ -0,0 +1,97 @@ +import type { DEFAULT_PERSONAL_CATEGORY } from "../adapter/storage/personal-key-store.ts"; +import type { RetentionInput } from "../contracts/duration.ts"; +import type { AppPersonalCategory } from "../register/index.ts"; + +/** + * The lawful basis of processing a category of personal data, as article 6(1) of the GDPR names + * them. `legal-obligation` keeps a category past `forget` for its retention, restricted, since the + * law that requires the data outweighs the request to erase it (article 17(3)(b)). + */ +export type LegalBasis = + | "consent" + | "contract" + | "legal-obligation" + | "vital-interests" + | "public-task" + | "legitimate-interests"; + +/** + * Every lawful basis, in the order of article 6(1). + */ +export const LEGAL_BASES: readonly LegalBasis[] = [ + "consent", + "contract", + "legal-obligation", + "vital-interests", + "public-task", + "legitimate-interests", +]; + +/** + * A category of personal data as `app/personal-data.ts` declares it: why it is processed, and for + * how long after the person's last activity it is kept, after which the runtime forgets it, as + * `forget(person, { categories })` does, with `reason: "retention"`: a duration string or + * milliseconds, as the configuration's durations, or whole calendar months or years, `"6y"`, + * which end on the same day of the month, or at the end of the month's last day when it has no + * such day. + */ +export interface PersonalDataCategory { + readonly basis: LegalBasis; + readonly retention: RetentionInput; +} + +/** + * What `app/personal-data.ts` exports as `categories`: each category by its name, which + * `personal(schema, { category })` and `.personal({ category })` take. `default` is the category + * of a field marked without one, and cannot be declared. + */ +export type PersonalDataCategories = Readonly>; + +/** + * `app/personal-data.ts`. + */ +export interface PersonalDataModule { + readonly categories: PersonalDataCategories; +} + +/** + * A category of personal data `forget` takes and a forgetting names: one `app/personal-data.ts` + * declares, or `"default"`, that of the fields marked without one. + */ +export type PersonalCategoryName = AppPersonalCategory | typeof DEFAULT_PERSONAL_CATEGORY; + +/** + * What `forget(personId, options)` takes besides the person. + */ +export interface ForgetOptions { + /** + * Forgets these categories of the person and nothing else, as when they withdraw a consent: + * their keys go, the rest stays readable, and the person is not forgotten as a whole. Without + * it, every category is. A category whose basis is `legal-obligation` is kept, restricted, until + * its retention ends, either way. + */ + readonly categories?: readonly PersonalCategoryName[]; +} + +type CategoriesOf = Module extends { readonly categories: infer Categories } + ? Categories + : never; + +/** + * The names of the categories an `app/personal-data.ts` module declares, which the generator + * registers so `personal({ category })` and `forget` take only those. + */ +export type PersonalCategoriesOf = Extract, string>; + +/** + * `true` when an `app/personal-data.ts` module exports `categories` as Bounda reads them, and + * declares no category `default`; otherwise a message saying what to change, which the + * generator's `.bounda/types.ts` fails to compile on. + */ +export type DeclaresPersonalData = [CategoriesOf] extends [never] + ? "app/personal-data.ts must export categories, each with its basis and its retention" + : CategoriesOf extends PersonalDataCategories + ? typeof DEFAULT_PERSONAL_CATEGORY extends PersonalCategoriesOf + ? 'app/personal-data.ts declares "default", the category of fields marked without one; name it otherwise' + : true + : "app/personal-data.ts declares a category without a basis Bounda knows or a retention"; diff --git a/packages/core/src/modules/personal.test.ts b/packages/core/src/modules/personal.test.ts index dfae68d6..e371da34 100644 --- a/packages/core/src/modules/personal.test.ts +++ b/packages/core/src/modules/personal.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from "vitest"; import { z } from "zod"; +import { DEFAULT_PERSONAL_CATEGORY } from "../adapter/storage/personal-key-store.ts"; import { person, personal, personalMarkOf } from "./personal.ts"; describe("personal and person", () => { @@ -19,15 +20,30 @@ describe("personal and person", () => { it("mark the schema they return, found through any wrapper and never on the one they took", () => { const email = z.email(); - expect(personalMarkOf(personal(email))).toBe("personal"); - expect(personalMarkOf(personal(email).optional().nullable())).toBe("personal"); - expect(personalMarkOf(person(z.string()))).toBe("person"); - expect(personalMarkOf(person(z.string()).default("x").readonly())).toBe("person"); - expect(personalMarkOf(personal(email).transform((value) => value?.toLowerCase()))).toBe( - "personal", + const marked = { mark: "personal", category: DEFAULT_PERSONAL_CATEGORY }; + expect(personalMarkOf(personal(email))).toEqual(marked); + expect(personalMarkOf(personal(email).optional().nullable())).toEqual(marked); + expect(personalMarkOf(person(z.string()))).toEqual({ mark: "person" }); + expect(personalMarkOf(person(z.string()).default("x").readonly())).toEqual({ + mark: "person", + }); + expect(personalMarkOf(personal(email).transform((value) => value?.toLowerCase()))).toEqual( + marked, ); expect(personalMarkOf(z.string().pipe(z.email()))).toBeNull(); expect(personalMarkOf(email)).toBeNull(); expect(personalMarkOf(z.object({ email: personal(email) }))).toBeNull(); }); + + it("mark a personal field with the category it is given, the default one without", () => { + expect(personalMarkOf(personal(z.email(), { category: "profile" }).optional())).toEqual({ + mark: "personal", + category: "profile", + }); + expect(personalMarkOf(personal(z.email(), {}))).toEqual({ + mark: "personal", + category: DEFAULT_PERSONAL_CATEGORY, + }); + expect(DEFAULT_PERSONAL_CATEGORY).toBe("default"); + }); }); diff --git a/packages/core/src/modules/personal.ts b/packages/core/src/modules/personal.ts index 8e23d78a..10db662f 100644 --- a/packages/core/src/modules/personal.ts +++ b/packages/core/src/modules/personal.ts @@ -1,4 +1,6 @@ import type { z } from "zod"; +import { DEFAULT_PERSONAL_CATEGORY } from "../adapter/storage/personal-key-store.ts"; +import type { AppPersonalCategory } from "../register/index.ts"; declare const personalBrand: unique symbol; declare const personBrand: unique symbol; @@ -31,25 +33,41 @@ export interface CommandPersonalSchema readonly [personalBrand]: true; } +/** + * What `personal` takes besides the schema. + */ +export interface PersonalOptions { + /** + * One of the categories `app/personal-data.ts` declares: the field is sealed under the person's + * key of that category, kept for that category's retention and forgotten with it. Without it, + * the field is in the default category, `DEFAULT_PERSONAL_CATEGORY`. + */ + readonly category?: AppPersonalCategory; +} + /** * Marks a field of an event's payload, or of a process's state, as personal data: it is stored - * encrypted with the key of the person the event or the state names, and reads as `null` once - * that person is forgotten. Any schema works, an object or an array included, at the top level of - * the payload or the state. A query payload cannot use it; a command's has its own, - * `CommandPersonalFunction`. + * encrypted with the key of the person the event or the state names in the field's category, and + * reads as `null` once that category of that person is forgotten. Any schema works, an object or + * an array included, at the top level of the payload or the state. A query payload cannot use it; + * a command's has its own, `CommandPersonalFunction`. */ export interface PersonalFunction { - (schema: Schema): PersonalSchema; + (schema: Schema, options?: PersonalOptions): PersonalSchema; } /** * Marks a field of a command's payload as personal data: wherever the command is stored, scheduled * with `delay` or dead-lettered, the field is encrypted with the key of the person the command - * names, and `forget` cancels the command or removes its dead letter. A command dispatched at - * once stores nothing, so the mark changes nothing for it. Same placement rules as an event's. + * names in the field's category, and forgetting that category cancels the command or removes its + * dead letter. A command dispatched at once stores nothing, so the mark changes nothing for it. + * Same placement rules and options as an event's. */ export interface CommandPersonalFunction { - (schema: Schema): CommandPersonalSchema; + ( + schema: Schema, + options?: PersonalOptions, + ): CommandPersonalSchema; } /** @@ -62,12 +80,16 @@ export interface PersonFunction { (schema: Schema): PersonSchema; } -const PERSONAL = new WeakSet(); +// Each schema `personal` returned, with the category it was given. +const PERSONAL = new WeakMap(); const PERSON = new WeakSet(); -export const personal: PersonalFunction = (schema: Schema) => { +export const personal: PersonalFunction = ( + schema: Schema, + options?: PersonalOptions, +) => { const marked = schema.transform((value): z.output | null => value); - PERSONAL.add(marked); + PERSONAL.set(marked, options?.category ?? DEFAULT_PERSONAL_CATEGORY); return marked as PersonalSchema; }; @@ -77,7 +99,12 @@ export const person: PersonFunction = (schema: Schema) return marked as PersonSchema; }; -export type PersonalMark = "personal" | "person"; +/** + * How a field is marked: `personal` with the category it is sealed under, or `person`. + */ +export type PersonalMark = + | { readonly mark: "personal"; readonly category: string } + | { readonly mark: "person" }; export interface PersonalMarkOfFunction { (schema: z.ZodType): PersonalMark | null; @@ -94,8 +121,9 @@ interface Wrapper { export const personalMarkOf: PersonalMarkOfFunction = (schema) => { let current: z.ZodType | undefined = schema; while (current !== undefined) { - if (PERSONAL.has(current)) return "personal"; - if (PERSON.has(current)) return "person"; + const category = PERSONAL.get(current); + if (category !== undefined) return { mark: "personal", category }; + if (PERSON.has(current)) return { mark: "person" }; const def = current._zod.def as Wrapper; current = def.innerType ?? def.in; } diff --git a/packages/core/src/modules/policy.ts b/packages/core/src/modules/policy.ts index f1304147..17d7bb42 100644 --- a/packages/core/src/modules/policy.ts +++ b/packages/core/src/modules/policy.ts @@ -1,5 +1,6 @@ import type { DurationInput } from "../contracts/duration.ts"; import type { EmptyPayload } from "./payload.ts"; +import type { ForgetOptions } from "./personal-data.ts"; /** * The shape of a policy module. `on` overrides the event type derived from the file name and may @@ -48,8 +49,9 @@ export type PolicyHandlerArgs; /** - * `forget` in the arguments of a policy or process handler. + * `forget` in the arguments of a policy or process handler, which takes the options a client's + * `forget` takes. */ export interface ReactionForgetFunction { - (personId: string): Promise; + (personId: string, options?: ForgetOptions): Promise; } diff --git a/packages/core/src/modules/registry.ts b/packages/core/src/modules/registry.ts index b198446f..699ea413 100644 --- a/packages/core/src/modules/registry.ts +++ b/packages/core/src/modules/registry.ts @@ -11,6 +11,7 @@ import type { CommandEntry, CommandModule, RejectionCodeOf } from "./command.ts" import type { EventModules } from "./event.ts"; import type { Simplify, UnionToIntersection } from "./naming.ts"; import type { HasPayload, PayloadInputOf } from "./payload.ts"; +import type { PersonalDataModule } from "./personal-data.ts"; import type { PolicyEntry } from "./policy.ts"; import type { PortModules } from "./port.ts"; import type { ProcessEntry } from "./process.ts"; @@ -55,6 +56,10 @@ export interface Registry { * `app/audit.ts`, when the project has one. */ readonly audit?: AuditModule; + /** + * `app/personal-data.ts`, when the project has one. + */ + readonly personalData?: PersonalDataModule; } // One overload per kind of call, so a result is typed by whether `options` carries `delay`. The diff --git a/packages/core/src/modules/view.ts b/packages/core/src/modules/view.ts index 460d7877..c14fe931 100644 --- a/packages/core/src/modules/view.ts +++ b/packages/core/src/modules/view.ts @@ -1,4 +1,6 @@ +import { DEFAULT_PERSONAL_CATEGORY } from "../adapter/storage/personal-key-store.ts"; import type { Simplify } from "./naming.ts"; +import type { PersonalOptions } from "./personal.ts"; /** * Storage types a read-model field can have. @@ -31,6 +33,11 @@ export interface FieldDefinition< * Adapters store a marked column as any other. */ readonly mark: Mark; + /** + * The category of a field marked `personal`, `DEFAULT_PERSONAL_CATEGORY` unless it was given + * one; `null` for any other field. + */ + readonly category: string | null; readonly _value?: Value; } @@ -48,12 +55,16 @@ export interface Field< unique(): Field; index(): Field; /** - * Marks the field as personal data of the person the row's `person` field names: the export of - * that person carries it, and a query that reads rows of the read model is recorded in the - * access log. A read model with personal fields needs a `person` field and a projection of - * `PersonalDataForgotten`, or `.bounda/types.ts` does not compile. + * Marks the field as personal data of the person the row's `person` field names, in the + * category given, the default one without: the export of that person carries it with its + * category, and a query that reads rows of the read model is recorded in the access log. A read + * model with personal fields needs a `person` field and a projection of `PersonalDataForgotten`, + * or `.bounda/types.ts` does not compile. */ - personal(this: Field): Field; + personal( + this: Field, + options?: PersonalOptions, + ): Field; /** * Marks the field that holds the id of the person the row's personal fields belong to. A read * model has at most one; its rows are in that person's export. @@ -133,6 +144,7 @@ interface FieldFlags { readonly isUnique: boolean; readonly isIndexed: boolean; readonly mark: FieldMark | null; + readonly category: string | null; } const createField = < @@ -149,11 +161,17 @@ const createField = < isUnique: flags.isUnique, isIndexed: flags.isIndexed, mark: flags.mark as Mark, + category: flags.category, optional: () => createField({ ...flags, isOptional: true }), primaryKey: () => createField({ ...flags, isPrimaryKey: true }), unique: () => createField({ ...flags, isUnique: true }), index: () => createField({ ...flags, isIndexed: true }), - personal: () => createField({ ...flags, mark: "personal" }), + personal: (options?: PersonalOptions) => + createField({ + ...flags, + mark: "personal", + category: options?.category ?? DEFAULT_PERSONAL_CATEGORY, + }), person: () => createField({ ...flags, mark: "person" }), }); @@ -165,6 +183,7 @@ const field = (type: Type): Field { personId: `p-${index}`, eventId: `ledger-event-${String(index).padStart(4, "0")}`, forgottenAt: new Date(Date.UTC(2025, 0, 1, 0, 0, index)).toISOString(), + categories: null, reason: "requested" as const, })), ); @@ -909,3 +910,95 @@ describe("expectForgotten", () => { ); }); }); + +describe("expectForgotten by categories", () => { + const categorized = { + aggregates: { + customer: { + state: { initialState: {} }, + events: { + customerBilled: { + payload: ({ z, personal }: EventPayloadArgs) => + z.object({ + newsletter: personal(z.email(), { category: "marketing" }), + invoice: personal(z.string(), { category: "invoices" }), + }), + evolve: () => ({}), + }, + }, + commands: { + billCustomer: { + module: { + payload: ({ z }: PayloadArgs) => + z.object({ customerId: z.string(), newsletter: z.string(), invoice: z.string() }), + handler: ({ + command, + events, + }: { + command: { payload: { newsletter: string; invoice: string } }; + events: Events; + }) => [events.customerBilled?.(command.payload)], + }, + }, + }, + policies: {}, + processes: {}, + }, + }, + readModels: {}, + personalData: { + categories: { + marketing: { basis: "consent", retention: "730d" }, + invoices: { basis: "legal-obligation", retention: "6y" }, + }, + }, + } as unknown as Registry; + + const billed = async () => { + const { app } = await createTestApp({ registry: categorized }); + await app.commands.billCustomer?.({ + customerId: "c-1", + newsletter: "ana@example.com", + invoice: "INV-2026-0001", + }); + return app; + }; + + it("checks a withdrawn consent alone, and refuses a category the person still holds", async () => { + const app = await billed(); + await app.forget("c-1", { categories: ["marketing"] }); + await expect( + expectForgotten(app, "c-1", { categories: ["marketing"] }), + ).resolves.toBeUndefined(); + await expect( + expectForgotten(app, "c-1", { + categories: ["marketing"], + places: { mail: ["ana@example.com"] }, + }), + ).rejects.toBeInstanceOf(NotForgottenError); + await expect(expectForgotten(app, "c-1", { categories: ["invoices"] })).rejects.toThrow( + '"c-1" still holds a key in force of "invoices": forget those categories before expectForgotten', + ); + await app.stop(); + }); + + it("leaves out what a legal obligation keeps after the person is forgotten as a whole", async () => { + const app = await billed(); + const before = await app.exportPersonalData("c-1"); + await app.forget("c-1"); + await expect( + expectForgotten(app, "c-1", { places: { kept: ["INV-2026-0001"] } }), + ).resolves.toBeUndefined(); + await expect( + expectForgotten(app, "c-1", { + before, + categories: ["marketing"], + places: { kept: ["INV-2026-0001"] }, + }), + ).resolves.toBeUndefined(); + await expect( + expectForgotten(app, "c-1", { before, places: { kept: ["INV-2026-0001"] } }), + ).rejects.toBeInstanceOf(NotForgottenError); + await app.stop(); + }); +}); diff --git a/packages/core/src/testing/expect-forgotten.ts b/packages/core/src/testing/expect-forgotten.ts index 80181410..3c89d602 100644 --- a/packages/core/src/testing/expect-forgotten.ts +++ b/packages/core/src/testing/expect-forgotten.ts @@ -1,9 +1,11 @@ import { type Adapter, isAdapter } from "../adapter/adapter.ts"; import type { ForgetLedger, ForgetLedgerEntry } from "../adapter/storage/forget-ledger.ts"; import type { InspectedStore, InspectedTable } from "../adapter/storage/inspection.ts"; +import { DEFAULT_PERSONAL_CATEGORY } from "../adapter/storage/personal-key-store.ts"; import type { BoundaClient } from "../kernel/app.ts"; -import { unseal } from "../kernel/personal-data/crypto.ts"; +import { categoryOf, unseal } from "../kernel/personal-data/crypto.ts"; import type { PersonalDataExport } from "../kernel/personal-data/export.ts"; +import type { PersonalCategoryName } from "../modules/personal-data.ts"; import { type Leaf, leavesIn, matches, needlesOf } from "./search.ts"; import type { TelemetryRecording } from "./telemetry.ts"; import type { TestWitness } from "./witness.ts"; @@ -81,6 +83,13 @@ export interface ExpectForgottenOptions { * was sent, a log of your own. */ readonly places?: Readonly>; + /** + * Checks only these categories, as after `forget(personId, { categories })`: what was sealed or + * kept of the person in them, and the fields of `before` in them, are looked for, and the + * person must hold no key in force of them. Without it, every category the person no longer + * holds a key of. + */ + readonly categories?: readonly PersonalCategoryName[]; } /** @@ -140,12 +149,32 @@ const ledgerEntries = async (ledger: ForgetLedger): Promise [ - ...exported.events.map(({ data }) => data), - ...exported.scheduledCommands.map(({ data }) => data), - ...exported.deadLetters.map(({ data }) => data), - ...Object.values(exported.readModels).flatMap((rows) => rows.map(({ data }) => data)), -]; +interface ExportedRecord { + readonly data: Readonly>; + readonly categories: Readonly>; +} + +// The fields of each record, of the categories asked for alone when they are. +const exportedValues = ( + exported: PersonalDataExport, + categories: readonly string[] | undefined, +): readonly unknown[] => { + const records: readonly ExportedRecord[] = [ + ...exported.events, + ...exported.scheduledCommands, + ...exported.deadLetters, + ...Object.values(exported.readModels).flat(), + ]; + return records.map(({ data, categories: of }) => + categories === undefined + ? data + : Object.fromEntries( + Object.entries(data).filter(([field]) => + categories.includes(of[field] ?? DEFAULT_PERSONAL_CATEGORY), + ), + ), + ); +}; const rowOf = (table: InspectedTable, row: Readonly>, index: number) => table.key.length === 0 @@ -212,10 +241,18 @@ const STORE_ADVICE: Readonly>> = { "A subscriber's last failure quotes the value in the clear: its event names the person by no marked field holding a value, so mark the field the value comes from personal(), and keep personal data out of error messages.", }; -const opened = async (witness: TestWitness, personId: string, held: string | null) => { +const opened = async ( + witness: TestWitness, + personId: string, + held: ReadonlySet, + categories: readonly string[] | undefined, +) => { const values: unknown[] = []; for (const { envelope, key } of witness.sealedFor(personId)) { - if (envelope.key === held) continue; + if (held.has(envelope.key)) continue; + if (categories !== undefined && !categories.includes(categoryOf({ $personal: envelope }))) { + continue; + } if (key === null) { throw new Error( `expectForgotten saw a value of "${personId}" sealed under a key the test app never held; report this as a bug in Bounda`, @@ -226,14 +263,30 @@ const opened = async (witness: TestWitness, personId: string, held: string | nul return values; }; -// The fingerprint of the key the person holds now: what was sealed under it is theirs still. -const heldKey = (tables: readonly InspectedTable[], personId: string): string | null => { - for (const table of tables) { - if (table.store !== "personalKeys") continue; - const row = table.rows.find((candidate) => candidate.person === personId); - if (row !== undefined) return String(row.fingerprint); - } - return null; +interface HeldKeys { + /** + * The fingerprints of every key the person holds, in force or kept past a forgetting: what was + * sealed under one is theirs still, or kept lawfully. + */ + readonly fingerprints: ReadonlySet; + /** + * The categories of the keys they hold in force. + */ + readonly inForce: ReadonlySet; +} + +const heldKeys = (tables: readonly InspectedTable[], personId: string): HeldKeys => { + const rows = tables + .filter((table) => table.store === "personalKeys") + .flatMap((table) => table.rows.filter((row) => row.person === personId)); + return { + fingerprints: new Set(rows.map((row) => String(row.fingerprint))), + inForce: new Set( + rows + .filter((row) => row.retained_since === null || row.retained_since === undefined) + .map((row) => String(row.category ?? DEFAULT_PERSONAL_CATEGORY)), + ), + }; }; /** @@ -243,8 +296,9 @@ const heldKey = (tables: readonly InspectedTable[], personId: string): string | * * Given the app `createTestApp` returned, it looks for every value of a field marked * `personal()` that the app sealed under a key the person no longer holds, and for every value a - * read model kept in a field marked `.personal()` for them before they were forgotten, and adds - * `values` and `before`. It searches each table of the storage and of every read model, read as + * read model kept in a field marked `.personal()` for them before its category was forgotten, and + * adds `values` and `before`; with `categories`, those of the categories named alone. A key a + * legal obligation kept past the forgetting is still held, so what it sealed is not looked for. It searches each table of the storage and of every read model, read as * stored and never opened, the forget ledger, every entry the app logged and, with `telemetry`, * every span. Given an adapter, such as one over a Durable Object's storage, it searches that * adapter's tables for `values` and `before` alone. @@ -274,17 +328,27 @@ export const expectForgotten: ExpectForgottenFunction = async (target, personId, "expectForgotten found no table to search: the adapter's database holds none of Bounda's, so it may be another one, or closed", ); } - const held = heldKey(tables, personId); + // Names, whatever the project registers, so any category read from the store compares. + const categories: readonly string[] | undefined = options.categories; + const held = heldKeys(tables, personId); + const stillHeld = [...held.inForce].filter( + (category) => categories === undefined || categories.includes(category), + ); + if (categories !== undefined && stillHeld.length > 0) { + throw new Error( + `"${personId}" still holds a key in force of ${stillHeld.map((category) => `"${category}"`).join(", ")}: forget those categories before expectForgotten`, + ); + } const values: unknown[] = [...(options.values ?? [])]; - if (options.before !== undefined) values.push(...exportedValues(options.before)); + if (options.before !== undefined) values.push(...exportedValues(options.before, categories)); if (apps !== undefined) { - values.push(...(await opened(apps.witness, personId, held))); - values.push(...apps.witness.keptFor(personId)); + values.push(...(await opened(apps.witness, personId, held.fingerprints, categories))); + values.push(...apps.witness.keptFor(personId, categories)); } const needles = needlesOf({ values, personId }); if (needles.length === 0) { throw new Error( - held === null + stillHeld.length === 0 ? `expectForgotten found no value of "${personId}" to look for: pass the ones the test gave them as values, or their export as before` : `"${personId}" still holds a key: forget them before expectForgotten, which looks only for what was sealed under keys they no longer hold`, ); diff --git a/packages/core/src/testing/witness.test.ts b/packages/core/src/testing/witness.test.ts index dc7a6ff0..c6614591 100644 --- a/packages/core/src/testing/witness.test.ts +++ b/packages/core/src/testing/witness.test.ts @@ -79,7 +79,7 @@ describe("the witness of a test app's storage", () => { payload: { email: sealed("c-1", "two", "fp-other") }, } as never); }); - const key = { fingerprint: "fp-c-1", cryptoKey: {} as CryptoKey }; + const key = { fingerprint: "fp-c-1", cryptoKey: {} as CryptoKey, retained: false }; witness.onPersonalKey("c-1", key); expect(witness.sealedFor("c-1")).toEqual([ { envelope: sealed("c-1", "one").$personal, key }, diff --git a/packages/core/src/testing/witness.ts b/packages/core/src/testing/witness.ts index 499feb16..d26c91e8 100644 --- a/packages/core/src/testing/witness.ts +++ b/packages/core/src/testing/witness.ts @@ -6,7 +6,10 @@ import type { Storage, StorageTransaction, } from "../adapter/adapter.ts"; -import type { PersonalEnvelope } from "../adapter/storage/personal-key-store.ts"; +import { + DEFAULT_PERSONAL_CATEGORY, + type PersonalEnvelope, +} from "../adapter/storage/personal-key-store.ts"; import type { Table } from "../adapter/storage/table.ts"; import type { LogFields, Logger } from "../contracts/logger.ts"; import { isSealed, type PersonalKey } from "../kernel/personal-data/crypto.ts"; @@ -21,15 +24,22 @@ export interface LoggedEntry { } /** - * A value a read model kept in a field marked `.personal()`, and how many times its person had - * been forgotten when it was written: only what was written before their last forgetting is - * theirs no longer. + * A value a read model kept in a field marked `.personal()`, its category, and how many times its + * person had been forgotten when it was written: only what was written before a later forgetting + * of its category is theirs no longer. */ interface KeptValue { readonly value: unknown; + readonly category: string; readonly forgettings: number; } +/** + * The categories a `PersonalDataForgotten` forgot, `null` for every one, as one recorded before + * categories did. + */ +type Forgetting = readonly string[] | null; + /** * Everything the test apps over one storage saw that `expectForgotten` needs once the person's * key is gone: the values sealed under each key, the keys, what the read models kept of each @@ -44,10 +54,10 @@ export interface TestWitness { */ sealedFor(person: string): readonly SealedFor[]; /** - * What the read models kept in their personal fields of the person, before they were last - * forgotten. + * What the read models kept in their personal fields of the person, in a category forgotten + * since; with `categories`, in one of those alone. */ - keptFor(person: string): readonly unknown[]; + keptFor(person: string, categories?: readonly string[]): readonly unknown[]; /** * `adapter`, observed: the same object for the same adapter, so the app still tells it apart * from another, as the forget ledger's check that it is not the storage does. @@ -90,7 +100,7 @@ const createTestWitness = (): TestWitness => { const keys = new Map>(); const envelopes = new Map>(); const kept = new Map(); - const forgettings = new Map(); + const forgettings = new Map(); const logs: LoggedEntry[] = []; const wrapped = new WeakMap(); @@ -115,7 +125,11 @@ const createTestWitness = (): TestWitness => { record.aggregateType === BOUNDA_AGGREGATE && typeof record.aggregateId === "string" ) { - forgettings.set(record.aggregateId, (forgettings.get(record.aggregateId) ?? 0) + 1); + const forgot: unknown = Reflect.get(Object(record.payload), "categories"); + forgettings.set(record.aggregateId, [ + ...(forgettings.get(record.aggregateId) ?? []), + Array.isArray(forgot) ? forgot.map(String) : null, + ]); } for (const [name, inner] of Object.entries(record)) observe(inner, name, seen); }; @@ -164,9 +178,13 @@ const createTestWitness = (): TestWitness => { const person = marks.person === null ? undefined : row[marks.person]; if (typeof person !== "string") return; const values = kept.get(person) ?? []; - for (const field of marks.personal) { - if (row[field] === undefined || row[field] === null) continue; - values.push({ value: row[field], forgettings: forgettings.get(person) ?? 0 }); + for (const { name, category } of marks.personal) { + if (row[name] === undefined || row[name] === null) continue; + values.push({ + value: row[name], + category, + forgettings: forgettings.get(person)?.length ?? 0, + }); } kept.set(person, values); }; @@ -183,7 +201,7 @@ const createTestWitness = (): TestWitness => { }, update: async (where, patch) => { const changes = patch as Record; - if (marks.personal.some((field) => changes[field] !== undefined)) { + if (marks.personal.some(({ name }) => changes[name] !== undefined)) { for (const row of await table.findMany({ where })) { keep({ ...(row as Record), ...changes }, marks); } @@ -235,10 +253,16 @@ const createTestWitness = (): TestWitness => { envelope, key: keys.get(person)?.get(envelope.key) ?? null, })), - keptFor: (person) => - (kept.get(person) ?? []) - .filter((value) => value.forgettings < (forgettings.get(person) ?? 0)) - .map(({ value }) => value), + keptFor: (person, categories) => { + const since = forgettings.get(person) ?? []; + return (kept.get(person) ?? []) + .filter( + ({ category, forgettings: before }) => + (categories === undefined || categories.includes(category)) && + since.slice(before).some((forgot) => forgot === null || forgot.includes(category)), + ) + .map(({ value }) => value); + }, storage: (adapter) => { const existing = wrapped.get(adapter); if (existing !== undefined) return existing; @@ -264,13 +288,15 @@ const createTestWitness = (): TestWitness => { interface ReadModelMarks { readonly person: string | null; - readonly personal: readonly string[]; + readonly personal: readonly { readonly name: string; readonly category: string }[]; } const marksOf = (fields: FieldsRecord): ReadModelMarks => { const entries = Object.entries(fields); return { person: entries.find(([, field]) => field.mark === "person")?.[0] ?? null, - personal: entries.filter(([, field]) => field.mark === "personal").map(([name]) => name), + personal: entries + .filter(([, field]) => field.mark === "personal") + .map(([name, field]) => ({ name, category: field.category ?? DEFAULT_PERSONAL_CATEGORY })), }; }; diff --git a/packages/core/test-types/config.test-d.ts b/packages/core/test-types/config.test-d.ts index 696f49ef..1a455013 100644 --- a/packages/core/test-types/config.test-d.ts +++ b/packages/core/test-types/config.test-d.ts @@ -248,6 +248,8 @@ describe("defineConfig", () => { }); defineConfig({ storage: sqlite, ports, runtime: { personalData: { retention: "30d" } } }); defineConfig({ storage: sqlite, ports, runtime: { personalData: { retention: 86_400_000 } } }); + defineConfig({ storage: sqlite, ports, runtime: { personalData: { retention: "6y" } } }); + defineConfig({ storage: sqlite, ports, runtime: { personalData: { retention: "18mo" } } }); defineConfig({ storage: sqlite, ports, @@ -426,6 +428,7 @@ describe("defineConfig", () => { }>(); expectTypeOf().toEqualTypeOf<{ readonly personId: string; + readonly categories: readonly string[] | null; readonly eventId: string; readonly forgottenAt: string; readonly reason: PersonalDataForgottenReason; diff --git a/packages/core/test-types/fixtures/order-app-inferred/.bounda/register.d.ts b/packages/core/test-types/fixtures/order-app-inferred/.bounda/register.d.ts index fe493710..edb4ecbd 100644 --- a/packages/core/test-types/fixtures/order-app-inferred/.bounda/register.d.ts +++ b/packages/core/test-types/fixtures/order-app-inferred/.bounda/register.d.ts @@ -1,10 +1,11 @@ import type { registry } from "./registry.ts"; -import type { PortsConfig, TestPorts } from "./types.ts"; +import type { PersonalCategory, PortsConfig, TestPorts } from "./types.ts"; declare module "@bounda-dev/core/register" { interface Register { readonly registry: typeof registry; readonly ports: PortsConfig; readonly testPorts: TestPorts; + readonly personalCategories: PersonalCategory; } } diff --git a/packages/core/test-types/fixtures/order-app-inferred/.bounda/types.ts b/packages/core/test-types/fixtures/order-app-inferred/.bounda/types.ts index 0cf66a42..9a36e9eb 100644 --- a/packages/core/test-types/fixtures/order-app-inferred/.bounda/types.ts +++ b/packages/core/test-types/fixtures/order-app-inferred/.bounda/types.ts @@ -36,4 +36,6 @@ export type ReactionCommands = core.ReactionCommandsFacadeOf<{ readonly placeOrder: typeof import("../app/domain/order/commands/place-order.ts"); }>; +export type PersonalCategory = never; + export type Queries = core.QueriesFacadeOf>; diff --git a/packages/core/test-types/fixtures/order-app/.bounda/register.d.ts b/packages/core/test-types/fixtures/order-app/.bounda/register.d.ts index fe493710..edb4ecbd 100644 --- a/packages/core/test-types/fixtures/order-app/.bounda/register.d.ts +++ b/packages/core/test-types/fixtures/order-app/.bounda/register.d.ts @@ -1,10 +1,11 @@ import type { registry } from "./registry.ts"; -import type { PortsConfig, TestPorts } from "./types.ts"; +import type { PersonalCategory, PortsConfig, TestPorts } from "./types.ts"; declare module "@bounda-dev/core/register" { interface Register { readonly registry: typeof registry; readonly ports: PortsConfig; readonly testPorts: TestPorts; + readonly personalCategories: PersonalCategory; } } diff --git a/packages/core/test-types/fixtures/order-app/.bounda/registry.ts b/packages/core/test-types/fixtures/order-app/.bounda/registry.ts index e5c8f2a5..ddbf5ef6 100644 --- a/packages/core/test-types/fixtures/order-app/.bounda/registry.ts +++ b/packages/core/test-types/fixtures/order-app/.bounda/registry.ts @@ -31,6 +31,7 @@ import * as orderPayment from "../app/domain/order/processes/order-payment/index import * as orderPaymentOnOrderPaid from "../app/domain/order/processes/order-payment/on-order-paid.ts"; import type * as orderReminders from "../app/domain/order/reminders.ts"; import * as orderState from "../app/domain/order/state.ts"; +import * as personalData from "../app/personal-data.ts"; import * as customerDirectoryOnBoundaPersonalDataForgotten from "../app/read/customer-directory/projections/bounda/personal-data-forgotten.ts"; import * as customerDirectoryOnCustomerCustomerRegistered from "../app/read/customer-directory/projections/customer/customer-registered.ts"; import * as getCustomer from "../app/read/customer-directory/queries/get-customer.ts"; @@ -58,7 +59,7 @@ export const registry = { }, order: { state: orderState, - stateFingerprint: "6c9864790b9c4e6d", + stateFingerprint: "f82ec98c65f75680", events: { orderCancelled, orderPaid, orderPlaced }, upcasts: { orderPlaced: orderPlacedUpcasts }, ports: { @@ -106,4 +107,5 @@ export const registry = { }, }, audit, + personalData, } as const satisfies Registry; diff --git a/packages/core/test-types/fixtures/order-app/.bounda/types.ts b/packages/core/test-types/fixtures/order-app/.bounda/types.ts index 1d2aefca..12ce46bb 100644 --- a/packages/core/test-types/fixtures/order-app/.bounda/types.ts +++ b/packages/core/test-types/fixtures/order-app/.bounda/types.ts @@ -66,6 +66,9 @@ export type ReactionCommands = core.ReactionCommandsFacadeOf<{ type Expect = Check; +export type PersonalCategory = core.PersonalCategoriesOf; +export type PersonalDataDeclaration = Expect>; + export type CustomerDirectoryRow = core.RowOf; export type CustomerDirectoryFieldTypes = core.FieldTypesOf; export type CustomerDirectoryPorts = core.EmptyPayload; diff --git a/packages/core/test-types/fixtures/order-app/app/+types/personal-data.ts b/packages/core/test-types/fixtures/order-app/app/+types/personal-data.ts new file mode 100644 index 00000000..c78f8e7d --- /dev/null +++ b/packages/core/test-types/fixtures/order-app/app/+types/personal-data.ts @@ -0,0 +1,5 @@ +import type * as core from "@bounda-dev/core"; + +export declare namespace PersonalData { + type Categories = core.PersonalDataCategories; +} diff --git a/packages/core/test-types/fixtures/order-app/app/domain/order/order-placed.ts b/packages/core/test-types/fixtures/order-app/app/domain/order/order-placed.ts index 79383f6e..439bfbc0 100644 --- a/packages/core/test-types/fixtures/order-app/app/domain/order/order-placed.ts +++ b/packages/core/test-types/fixtures/order-app/app/domain/order/order-placed.ts @@ -6,7 +6,7 @@ export const payload = ({ z, personal, person }: Event.PayloadArgs) => lines: z.array( z.object({ sku: z.string(), quantity: z.int().positive(), unitPrice: z.number() }), ), - note: personal(z.string()).optional(), + note: personal(z.string(), { category: "delivery" }).optional(), }); export const evolve = ({ state, event }: Event.EvolveArgs) => ({ diff --git a/packages/core/test-types/fixtures/order-app/app/personal-data.ts b/packages/core/test-types/fixtures/order-app/app/personal-data.ts new file mode 100644 index 00000000..5870cd8b --- /dev/null +++ b/packages/core/test-types/fixtures/order-app/app/personal-data.ts @@ -0,0 +1,6 @@ +import type { PersonalData } from "./+types/personal-data"; + +export const categories = { + delivery: { basis: "contract", retention: "30d" }, + invoices: { basis: "legal-obligation", retention: "6y" }, +} satisfies PersonalData.Categories; diff --git a/packages/core/test-types/inference.test-d.ts b/packages/core/test-types/inference.test-d.ts index cd198cf9..fd86e079 100644 --- a/packages/core/test-types/inference.test-d.ts +++ b/packages/core/test-types/inference.test-d.ts @@ -7,6 +7,7 @@ import type { BoundaClient, CancelledCommand, CatchUpReadModelsArgs, + CategoryRetentionStatus, CausationContext, CheckActorFunction, Clock, @@ -32,6 +33,8 @@ import type { EnvSection, EventMetadata, EventPayloadArgs, + ExportedCategories, + ExportedCategory, ExportedData, ExportedDeadLetter, ExportedEvent, @@ -45,23 +48,29 @@ import type { FieldTypesOf, FieldTypesOfRow, ForgetLedgerReconciliationStatus, + ForgetOptions, ForgetsPersonalData, HasPersonalFields, ImplementationModule, Instant, KeyRotationStatus, + LegalBasis, Logger, OnPersonalDataAccess, OnPersonalDataAccessArgs, PayloadArgs, PayloadInputOf, + PersonalCategoryName, PersonalDataAccess, PersonalDataBackfillLag, + PersonalDataCategories, + PersonalDataCategory, PersonalDataExport, PersonalDataForgottenEvent, PersonalDataForgottenPayload, PersonalDataForgottenReason, PersonalFunction, + PersonalOptions, PersonFunction, ProcessCorrelation, ProcessingRestrictedEvent, @@ -79,6 +88,7 @@ import type { RejectFunction, RejectionCodeOf, RemovedDeadLetter, + RetentionInput, RetentionStatus, ScheduledDispatch, SchemaStatus, @@ -115,8 +125,11 @@ import type { OrderSummaryRow as GeneratedOrderSummaryRow, OrderSummaryFieldTypes, OrderSummaryForgetting, + PersonalCategory, + PersonalDataDeclaration, Queries, } from "./fixtures/order-app/.bounda/types.ts"; +import type { PersonalData } from "./fixtures/order-app/app/+types/personal-data.ts"; import type * as audit from "./fixtures/order-app/app/audit.ts"; import type { Event as CustomerRegistered } from "./fixtures/order-app/app/domain/customer/+types/customer-registered.ts"; import type { Command as RegisterCustomer } from "./fixtures/order-app/app/domain/customer/commands/+types/register-customer.ts"; @@ -427,7 +440,9 @@ describe("idempotency keys", () => { }); it("lets policies and processes forget a person with their run, and no command handler", () => { - expectTypeOf().toEqualTypeOf<(personId: string) => Promise>(); + expectTypeOf().toEqualTypeOf< + (personId: string, options?: ForgetOptions) => Promise + >(); expectTypeOf().toEqualTypeOf(); expectTypeOf< NotifyOnOrderPlaced.HandlerArgs["forget"] @@ -734,10 +749,15 @@ describe("personal data", () => { expectTypeOf().toEqualTypeOf<{ readonly personId: string; readonly reason: PersonalDataForgottenReason; + readonly whole: boolean; + readonly categories: readonly PersonalCategoryName[]; + readonly retained: readonly PersonalCategoryName[]; + readonly expired: readonly PersonalCategoryName[]; readonly fingerprints?: readonly string[]; readonly cancelledCommands: readonly CancelledCommand[]; readonly removedDeadLetters: readonly RemovedDeadLetter[]; }>(); + expectTypeOf().toEqualTypeOf<"default" | "delivery" | "invoices">(); expectTypeOf().toEqualTypeOf<"requested" | "retention">(); expectTypeOf< AuditOnPersonalDataForgotten.HandlerArgs["event"]["payload"]["reason"] @@ -767,10 +787,41 @@ describe("personal data", () => { }>(); }); + it("names the categories app/personal-data.ts declares, and checks the declaration", () => { + expectTypeOf().toEqualTypeOf<"delivery" | "invoices">(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf<{ + readonly basis: LegalBasis; + readonly retention: RetentionInput; + }>(); + expectTypeOf().toEqualTypeOf(); + expectTypeOf().toEqualTypeOf< + | "consent" + | "contract" + | "legal-obligation" + | "vital-interests" + | "public-task" + | "legitimate-interests" + >(); + expectTypeOf[1]>().toEqualTypeOf< + PersonalOptions | undefined + >(); + expectTypeOf().toEqualTypeOf<{ + readonly category?: "delivery" | "invoices"; + }>(); + expectTypeOf().toEqualTypeOf< + string | null | undefined + >(); + }); + it("lets every client forget a person, answering like a command", () => { expectTypeOf().toEqualTypeOf< - (personId: string) => Promise + (personId: string, options?: ForgetOptions) => Promise >(); + expectTypeOf().toEqualTypeOf<{ + readonly categories?: readonly ("default" | "delivery" | "invoices")[]; + }>(); expectTypeOf().toEqualTypeOf(); }); @@ -784,6 +835,7 @@ describe("personal data", () => { expectTypeOf().toEqualTypeOf<{ readonly personId: string; readonly exportedAt: string; + readonly categories: Readonly>; readonly events: readonly ExportedEvent[]; readonly scheduledCommands: readonly ExportedScheduledCommand[]; readonly deadLetters: readonly ExportedDeadLetter[]; @@ -800,6 +852,7 @@ describe("personal data", () => { expectTypeOf().toEqualTypeOf<{ readonly key: unknown; readonly data: ExportedData; + readonly categories: ExportedCategories; }>(); expectTypeOf().toEqualTypeOf<{ readonly id: string; @@ -810,12 +863,14 @@ describe("personal data", () => { readonly position: number; readonly timestamp: string; readonly data: ExportedData; + readonly categories: ExportedCategories; }>(); expectTypeOf().toEqualTypeOf<{ readonly commandId: string; readonly type: string; readonly executeAt: string; readonly data: ExportedData; + readonly categories: ExportedCategories; }>(); expectTypeOf().toEqualTypeOf<{ readonly id: string; @@ -824,8 +879,15 @@ describe("personal data", () => { readonly errorMessage?: string | null; readonly errorStack?: string | null; readonly data: ExportedData; + readonly categories: ExportedCategories; }>(); expectTypeOf().toEqualTypeOf>>(); + expectTypeOf().toEqualTypeOf>>(); + expectTypeOf().toEqualTypeOf<{ + readonly basis: LegalBasis | null; + readonly retention: string | null; + readonly restrictedUntil: string | null; + }>(); }); it("lets every client restrict a person's processing and lift it, answering like a command", () => { @@ -1350,7 +1412,12 @@ describe("operations", () => { readonly completeAt: string | null; }>(); expectTypeOf().toEqualTypeOf<{ - readonly retentionMs: number; + readonly categories: readonly CategoryRetentionStatus[]; + readonly keptBeyond: number; + }>(); + expectTypeOf().toEqualTypeOf<{ + readonly category: string; + readonly retention: string; readonly due: number; readonly nextDueAt: string | null; }>(); diff --git a/packages/core/test-types/negatives.test-d.ts b/packages/core/test-types/negatives.test-d.ts index 6d627285..720c71b3 100644 --- a/packages/core/test-types/negatives.test-d.ts +++ b/packages/core/test-types/negatives.test-d.ts @@ -1,10 +1,13 @@ import { + type BoundaClient, + type DeclaresPersonalData, DomainError, type EventPayloadArgs, type FieldsArgs, type ForgetsPersonalData, type ImplementationModule, type Instant, + type PersonalDataCategory, type PolicyModule, type ProcessDeadlineField, type ProcessHandlerReturnCheck, @@ -243,6 +246,46 @@ describe("what does not compile", () => { void marks; }); + it("a category app/personal-data.ts does not declare, or a declaration Bounda cannot read", () => { + const marks = ({ z, personal }: EventPayloadArgs) => + z.object({ + note: personal(z.string(), { category: "delivery" }), + // @ts-expect-error a misspelled category does not compile + invoice: personal(z.string(), { category: "invoice" }), + }); + void marks; + const fields = ({ f }: FieldsArgs) => ({ + // @ts-expect-error nor in a read model's view + note: f.string().personal({ category: "deliveries" }), + }); + void fields; + const forget = (app: BoundaClient) => [ + app.forget("c-1", { categories: ["default", "invoices"] }), + // @ts-expect-error nor in forget + app.forget("c-1", { categories: ["profile"] }), + ]; + void forget; + type Expect = Check; + const year: PersonalDataCategory = { basis: "legal-obligation", retention: "6y" }; + // @ts-expect-error a retention takes days, months or years, never weeks + const weeks: PersonalDataCategory = { basis: "consent", retention: "2w" }; + void [year, weeks]; + type DeclaresDefault = { + readonly categories: { readonly default: { basis: "consent"; retention: "1d" } }; + }; + type UnknownBasis = { + readonly categories: { readonly profile: { basis: "whim"; retention: "1d" } }; + }; + // @ts-expect-error "default" is the category of fields marked without one + type Default = Expect>; + // @ts-expect-error a basis article 6 does not name + type Basis = Expect>; + // @ts-expect-error a module without categories + type Nothing = Expect>; + const checks: [Default?, Basis?, Nothing?] = []; + void checks; + }); + it("a read model that projects personal fields without handling forgetting", () => { type Expect = Check; type Unhandled = ForgetsPersonalData< diff --git a/packages/core/test-types/testing.test-d.ts b/packages/core/test-types/testing.test-d.ts index e4aa4273..dd03f116 100644 --- a/packages/core/test-types/testing.test-d.ts +++ b/packages/core/test-types/testing.test-d.ts @@ -100,7 +100,11 @@ describe("expectForgotten", () => { readonly before?: PersonalDataExport; readonly telemetry?: TelemetryRecording; readonly places?: Readonly>; + readonly categories?: readonly ("default" | "delivery" | "invoices")[]; }>(); + // @ts-expect-error a category app/personal-data.ts does not declare + const misspelled: ExpectForgottenOptions = { categories: ["deliveries"] }; + void misspelled; // @ts-expect-error before is an export, not a list of values const wrong: ExpectForgottenOptions = { before: ["ana@example.com"] }; void wrong; diff --git a/packages/postgresql/src/dead-letter-store.ts b/packages/postgresql/src/dead-letter-store.ts index d4bc6413..c97a2e3e 100644 --- a/packages/postgresql/src/dead-letter-store.ts +++ b/packages/postgresql/src/dead-letter-store.ts @@ -15,7 +15,7 @@ import { postgresqlDialect, } from "@bounda-dev/core/adapter/sql"; import type { PostgresqlDatabase } from "./database.ts"; -import { naming } from "./held-keys.ts"; +import { naming, outsideKept } from "./held-keys.ts"; export interface CreatePostgresqlDeadLetterStoreArgs { readonly db: PostgresqlDatabase; @@ -163,8 +163,13 @@ export const createPostgresqlDeadLetterStore: CreatePostgresqlDeadLetterStoreFun [person], ) ).map(toLetter), - forget: async (person) => - (await db.all(`DELETE FROM ${table} WHERE ${naming("$1")} RETURNING ${COLUMNS}`, [person])) + forget: async (person, keep = []) => + ( + await db.all( + `DELETE FROM ${table} WHERE ${naming("$1")} AND ${outsideKept({ person: "$1", keep: "$2" })} RETURNING ${COLUMNS}`, + [person, postgresqlDialect.encode("json", keep)], + ) + ) .map(toLetter) .sort(byFirstFailure), firstResolvedAt: async () => { diff --git a/packages/postgresql/src/held-keys.ts b/packages/postgresql/src/held-keys.ts index 2d274ab4..db120d79 100644 --- a/packages/postgresql/src/held-keys.ts +++ b/packages/postgresql/src/held-keys.ts @@ -10,13 +10,30 @@ export interface HeldKeysFunction { /** * A condition true when every person in `persons` named with a fingerprint holds, in - * `personalKeys`, the key of that fingerprint. A person named with `null` was read in the clear - * and needs no key. A person named under two keys never holds both. The keys it finds stay locked - * until the transaction ends: a forget that removes one waits for the write, and then finds the - * row to remove. + * `personalKeys`, the key of that fingerprint in force. A person named with `null` was read in the + * clear and needs no key. Each entry names one key, which a person holds at most once. The keys it + * finds stay locked until the transaction ends: a forget that removes or retains one waits for the + * write, and then finds the row. */ export const heldKeys: HeldKeysFunction = ({ personalKeys, persons }) => - `(SELECT count(*) FROM (SELECT 1 FROM ${personalKeys} AS "k" JOIN jsonb_to_recordset(${persons}::jsonb) AS "p"("person" text, "fingerprint" text) ON "k"."person" = "p"."person" AND "k"."fingerprint" = "p"."fingerprint" FOR SHARE OF "k") AS "held") = (SELECT count(*) FROM jsonb_to_recordset(${persons}::jsonb) AS "p"("person" text, "fingerprint" text) WHERE "p"."fingerprint" IS NOT NULL)`; + `(SELECT count(*) FROM (SELECT 1 FROM ${personalKeys} AS "k" JOIN jsonb_to_recordset(${persons}::jsonb) AS "p"("person" text, "fingerprint" text) ON "k"."person" = "p"."person" AND "k"."fingerprint" = "p"."fingerprint" WHERE "k"."retained_since" IS NULL FOR SHARE OF "k") AS "held") = (SELECT count(*) FROM jsonb_to_recordset(${persons}::jsonb) AS "p"("person" text, "fingerprint" text) WHERE "p"."fingerprint" IS NOT NULL)`; + +export interface OutsideKeptArgs { + // The placeholders of the person and of the JSON array of fingerprints kept. + readonly person: string; + readonly keep: string; +} + +export interface OutsideKeptFunction { + (args: OutsideKeptArgs): string; +} + +/** + * A condition true for a row whose `persons` names `person` under a key outside `keep`, or read in + * the clear. + */ +export const outsideKept: OutsideKeptFunction = ({ person, keep }) => + `EXISTS (SELECT 1 FROM jsonb_array_elements("persons") AS "n" WHERE "n"->>'person' = ${person}::text AND ("n"->>'fingerprint' IS NULL OR "n"->>'fingerprint' NOT IN (SELECT jsonb_array_elements_text(${keep}::jsonb))))`; export interface NamingFunction { (person: string): string; diff --git a/packages/postgresql/src/idempotency-store.ts b/packages/postgresql/src/idempotency-store.ts index 2cb5467d..a5a5e3e4 100644 --- a/packages/postgresql/src/idempotency-store.ts +++ b/packages/postgresql/src/idempotency-store.ts @@ -7,7 +7,7 @@ import type { } from "@bounda-dev/core/adapter"; import { postgresqlDialect } from "@bounda-dev/core/adapter/sql"; import type { PostgresqlDatabase } from "./database.ts"; -import { heldKeys, naming } from "./held-keys.ts"; +import { heldKeys, naming, outsideKept } from "./held-keys.ts"; export interface CreatePostgresqlIdempotencyStoreArgs { readonly db: PostgresqlDatabase; @@ -79,6 +79,10 @@ export const createPostgresqlIdempotencyStore: CreatePostgresqlIdempotencyStoreF ]); return rows.length; }, - forget: (person) => db.run(`DELETE FROM ${table} WHERE ${naming("$1")}`, [person]), + forget: (person, keep = []) => + db.run( + `DELETE FROM ${table} WHERE ${naming("$1")} AND ${outsideKept({ person: "$1", keep: "$2" })}`, + [person, postgresqlDialect.encode("json", keep)], + ), }; }; diff --git a/packages/postgresql/src/index.ts b/packages/postgresql/src/index.ts index 65a5a846..0da67cfb 100644 --- a/packages/postgresql/src/index.ts +++ b/packages/postgresql/src/index.ts @@ -169,6 +169,7 @@ export const postgresql: PostgresqlFunction = (options) => { dialect: postgresqlDialect, db: database, table: tables.personalKeys, + persons: tables.persons, }), persons: createSqlPersonStore({ dialect: postgresqlDialect, diff --git a/packages/postgresql/src/migrations.ts b/packages/postgresql/src/migrations.ts index f0039802..42102149 100644 --- a/packages/postgresql/src/migrations.ts +++ b/packages/postgresql/src/migrations.ts @@ -1,6 +1,7 @@ import { ConfigurationError } from "@bounda-dev/core"; import type { SchemaVersion } from "@bounda-dev/core/adapter"; import { + forgetLedgerCategoriesStatement, forgetLedgerMarksStatement, forgetLedgerReasonStatements, forgetLedgerStatements, @@ -137,6 +138,53 @@ const adoptStorageTables = async ( for (const statement of storageSchemaStatements(tables)) await tx.run(statement, []); }; +const keysIndex = (tables: StorageTables, suffix: string): string => + quoteIdentifier(`${tables.personalKeys.slice(1, -1)}_${suffix}_idx`); + +/** + * The third migration: a key per person and category instead of one per person, each named by + * its fingerprint, with the person's last activity on it and whether a forgetting kept it under a + * legal obligation. Every key there was is the person's key in the default category, last active + * when the person was, unless they were forgotten after that, when the key came later. Every + * forgetting recorded before forgot the person as a whole, in the default category. + */ +const adoptCategories = async ( + tx: SqlTransaction, + schema: string, + tables: StorageTables, +): Promise => { + const keys = tables.personalKeys; + const forgettings = `UPDATE ${tables.events} SET "payload" = "payload" || '{"categories":["default"],"retained":[],"whole":true}'::jsonb WHERE "aggregate_type" = 'bounda' AND "type" = 'PersonalDataForgotten' AND "payload"->'categories' IS NULL`; + // A schema migrated before, whose versions were lost, has the keys as this leaves them. + if ((await columnsOf(tx, schema, keys)).has("category")) { + await tx.run(forgettings, []); + return; + } + const [primary] = await tx.all( + `SELECT "constraint_name" FROM information_schema.table_constraints WHERE "table_schema" = $1 AND "table_name" = $2 AND "constraint_type" = 'PRIMARY KEY'`, + [schema, keys.slice(1, -1)], + ); + const statements = [ + `ALTER TABLE ${keys} ADD COLUMN "category" text NOT NULL DEFAULT 'default', ADD COLUMN "last_active_at" text, ADD COLUMN "retained_since" text, ADD COLUMN "forget_held_until" text, ADD COLUMN "forget_attempts" integer NOT NULL DEFAULT 0`, + `ALTER TABLE ${keys} ALTER COLUMN "category" DROP DEFAULT`, + ...(primary === undefined + ? [] + : [ + `ALTER TABLE ${keys} DROP CONSTRAINT ${quoteIdentifier(String(primary.constraint_name))}`, + ]), + `ALTER TABLE ${keys} ADD PRIMARY KEY ("person", "fingerprint")`, + `UPDATE ${keys} AS "k" SET "last_active_at" = "p"."last_active_at" FROM ${tables.persons} AS "p" WHERE "p"."person" = "k"."person" AND "p"."forgotten_at" IS NULL`, + // One key in force per person and category, which `add` resolves its race on. + `CREATE UNIQUE INDEX ${keysIndex(tables, "in_force")} ON ${keys} ("person", "category") WHERE "retained_since" IS NULL`, + // What a category's retention reads, least recently active first. + `CREATE INDEX ${keysIndex(tables, "due")} ON ${keys} ("category", "last_active_at") WHERE "last_active_at" IS NOT NULL`, + // The keys a forgetting kept, which no legal obligation may keep any more. + `CREATE INDEX ${keysIndex(tables, "retained")} ON ${keys} ("retained_since") WHERE "retained_since" IS NOT NULL`, + forgettings, + ]; + for (const statement of statements) await tx.run(statement, []); +}; + export interface PostgresqlStorageScopeArgs { readonly schema: string; readonly tables: StorageTables; @@ -154,6 +202,7 @@ export const postgresqlStorageScope: PostgresqlStorageScopeFunction = ({ schema, version: 2, migrate: (tx) => tx.run(subscriberFailuresTableStatement(tables.subscriberFailures), []), }, + { version: 3, migrate: (tx) => adoptCategories(tx, schema, tables) }, ], }); @@ -188,6 +237,10 @@ export const postgresqlForgetLedgerScope: PostgresqlForgetLedgerScopeFunction = for (const statement of forgetLedgerReasonStatements(table)) await tx.run(statement, []); }, }, + { + version: 4, + migrate: (tx) => tx.run(forgetLedgerCategoriesStatement(table, postgresqlDialect), []), + }, ], }); diff --git a/packages/postgresql/src/postgresql.test.ts b/packages/postgresql/src/postgresql.test.ts index 8eacf44b..8df8c0cd 100644 --- a/packages/postgresql/src/postgresql.test.ts +++ b/packages/postgresql/src/postgresql.test.ts @@ -1,6 +1,7 @@ import { ConfigurationError, createApp, + DEFAULT_PERSONAL_CATEGORY, type EventPayloadArgs, type FieldsArgs, fieldBuilder as f, @@ -43,8 +44,13 @@ import { afterAll, describe, expect, it, vi } from "vitest"; import { createPostgresqlDatabase } from "./database.ts"; import { createPostgresqlEventStore } from "./event-store.ts"; import { type PostgresqlAdapter, type PostgresqlOptions, postgresql } from "./index.ts"; -import { migratePostgresql, postgresqlCheckpointsScope } from "./migrations.ts"; +import { + migratePostgresql, + postgresqlCheckpointsScope, + postgresqlStorageScope, +} from "./migrations.ts"; import { resolvePostgresqlOptions } from "./options.ts"; +import { storageTablesFor } from "./schema.ts"; const reuseContainer = process.env.BOUNDA_PG_REUSE === "1"; @@ -123,7 +129,7 @@ describe.skipIf(container === null)("postgresql adapter", () => { idempotencyStoreContract({ create: () => openStorage() }); snapshotStoreContract({ create: () => openStorage() }); storageTransactionContract({ create: () => openStorage() }); - personalKeyStoreContract({ create: async () => (await openStorage()).personalKeys }); + personalKeyStoreContract({ create: async () => openStorage() }); personStoreContract({ create: async () => (await openStorage()).persons }); inspectionContract({ create: async () => { @@ -368,6 +374,47 @@ describe.skipIf(container === null)("postgresql adapter", () => { ]); }); + it("takes in the keys of a schema from before categories, in the default category, last active when their person was", async () => { + const adapter = fresh(); + const { tablePrefix: prefix, schema } = resolvePostgresqlOptions(adapter.options); + const client = postgres(url, { max: 1, onnotice: () => undefined }); + closers.push(() => client.end({ timeout: 5 })); + const db = createPostgresqlDatabase(client); + const tables = storageTablesFor({ prefix, schema }); + const current = postgresqlStorageScope({ schema, tables }); + await migratePostgresql({ + db, + schema, + prefix, + scopes: [{ ...current, migrations: current.migrations.slice(0, 1) }], + }); + const at = "2026-01-01T00:00:00.000Z"; + await db.run( + `INSERT INTO "${schema}".${tables.persons} ("person", "last_active_at", "forgotten_at") VALUES ('p-1', $1, NULL), ('p-2', $1, $1)`, + [at], + ); + await db.run( + `INSERT INTO "${schema}".${tables.personalKeys} ("person", "key", "fingerprint") VALUES ('p-1', 'k-1', 'f-1'), ('p-2', 'k-2', 'f-2'), ('p-3', 'k-3', 'f-3')`, + [], + ); + const storage = await openStorage(adapter); + expect(await storage.personalKeys.find("p-1", "f-1")).toEqual({ + category: DEFAULT_PERSONAL_CATEGORY, + key: { key: "k-1", fingerprint: "f-1", kid: null }, + retainedSince: null, + lastActiveAt: at, + }); + expect((await storage.personalKeys.find("p-2", "f-2"))?.lastActiveAt).toBeNull(); + expect((await storage.personalKeys.find("p-3", "f-3"))?.lastActiveAt).toBeNull(); + expect( + await storage.personalKeys.add("p-1", "invoices", { + key: "k-4", + fingerprint: "f-4", + kid: null, + }), + ).toEqual({ key: "k-4", fingerprint: "f-4", kid: null }); + }); + it("creates the forget ledger's table in its schema once, and takes no lock once it is there", async () => { const schema = `ledger_${run}`; const adapter = fresh({ schema }); @@ -491,7 +538,11 @@ describe.skipIf(container === null)("postgresql adapter", () => { it("holds off a writer of the rows an append checked, that takes no append lock, until the append commits", async () => { const storage = await openStorage(); - await storage.personalKeys.add("c-1", { key: "k", fingerprint: "f", kid: null }); + await storage.personalKeys.add("c-1", DEFAULT_PERSONAL_CATEGORY, { + key: "k", + fingerprint: "f", + kid: null, + }); await storage.persons.restrict("c-2", new Date()); await storage.persons.unrestrict("c-2"); const steps: string[] = []; @@ -518,7 +569,7 @@ describe.skipIf(container === null)("postgresql adapter", () => { }); await checking; const writes = [ - storage.personalKeys.remove("c-1").then(() => steps.push("key removed")), + storage.personalKeys.remove("c-1", "f").then(() => steps.push("key removed")), storage.persons.restrict("c-2", new Date()).then(() => steps.push("restricted")), ]; // A writer waits on a row lock the append holds (the pool may queue the other), and nothing diff --git a/packages/postgresql/src/scheduler.ts b/packages/postgresql/src/scheduler.ts index dbe56c9a..24a73731 100644 --- a/packages/postgresql/src/scheduler.ts +++ b/packages/postgresql/src/scheduler.ts @@ -8,7 +8,7 @@ import type { } from "@bounda-dev/core/adapter"; import { byExecuteAt, earliestDue, postgresqlDialect } from "@bounda-dev/core/adapter/sql"; import type { PostgresqlDatabase } from "./database.ts"; -import { naming } from "./held-keys.ts"; +import { naming, outsideKept } from "./held-keys.ts"; export interface CreatePostgresqlSchedulerArgs { readonly db: PostgresqlDatabase; @@ -193,8 +193,13 @@ export const createPostgresqlScheduler: CreatePostgresqlSchedulerFunction = ({ d [person], ) ).map(toScheduled), - forget: async (person) => - (await db.all(`DELETE FROM ${table} WHERE ${naming("$1")} RETURNING ${COLUMNS}`, [person])) + forget: async (person, keep = []) => + ( + await db.all( + `DELETE FROM ${table} WHERE ${naming("$1")} AND ${outsideKept({ person: "$1", keep: "$2" })} RETURNING ${COLUMNS}`, + [person, postgresqlDialect.encode("json", keep)], + ) + ) .map(toScheduled) .sort(byExecuteAt), }; diff --git a/packages/postgresql/src/snapshot-store.ts b/packages/postgresql/src/snapshot-store.ts index 377ca230..705f5e38 100644 --- a/packages/postgresql/src/snapshot-store.ts +++ b/packages/postgresql/src/snapshot-store.ts @@ -1,6 +1,6 @@ import type { KeyedPerson, Snapshot, SnapshotStore } from "@bounda-dev/core/adapter"; import { postgresqlDialect, type SqlExecutor } from "@bounda-dev/core/adapter/sql"; -import { heldKeys, naming } from "./held-keys.ts"; +import { heldKeys, naming, outsideKept } from "./held-keys.ts"; export interface CreatePostgresqlSnapshotStoreArgs { readonly db: SqlExecutor; @@ -51,5 +51,9 @@ export const createPostgresqlSnapshotStore: CreatePostgresqlSnapshotStoreFunctio aggregateType, aggregateId, ]), - forget: (person) => db.run(`DELETE FROM ${table} WHERE ${naming("$1")}`, [person]), + forget: (person, keep = []) => + db.run( + `DELETE FROM ${table} WHERE ${naming("$1")} AND ${outsideKept({ person: "$1", keep: "$2" })}`, + [person, postgresqlDialect.encode("json", keep)], + ), }); diff --git a/packages/react-router/src/cloudflare/clients.test.ts b/packages/react-router/src/cloudflare/clients.test.ts index db307ac9..0e606408 100644 --- a/packages/react-router/src/cloudflare/clients.test.ts +++ b/packages/react-router/src/cloudflare/clients.test.ts @@ -76,6 +76,7 @@ describe("cloudflareClients", () => { await client.deadLetters.discard("d-2"); await client.rebuildReadModel("orders"); await client.forget("c-1"); + await client.forget("c-1", { categories: ["default"] }); await client.exportPersonalData("c-1"); await client.restrict("c-2"); await client.unrestrict("c-2"); @@ -89,7 +90,8 @@ describe("cloudflareClients", () => { ["acme", "retryDeadLetter", "d-1", undefined], ["acme", "discardDeadLetter", "d-2", undefined], ["acme", "rebuildReadModel", "orders"], - ["acme", "forget", "c-1", "read-your-writes", undefined], + ["acme", "forget", "c-1", "read-your-writes", undefined, undefined], + ["acme", "forget", "c-1", "read-your-writes", undefined, { categories: ["default"] }], ["acme", "exportPersonalData", "c-1", undefined], ["acme", "restrict", "c-2", "read-your-writes", undefined], ["acme", "unrestrict", "c-2", "read-your-writes", undefined], diff --git a/packages/sqlite/src/sqlite.test.ts b/packages/sqlite/src/sqlite.test.ts index 81671ad2..a4a0dbe7 100644 --- a/packages/sqlite/src/sqlite.test.ts +++ b/packages/sqlite/src/sqlite.test.ts @@ -49,7 +49,7 @@ describe("sqlite adapter in memory", () => { schedulerContract({ create: async () => (await openStorage()).scheduler }); snapshotStoreContract({ create: () => openStorage() }); storageTransactionContract({ create: () => openStorage() }); - personalKeyStoreContract({ create: async () => (await openStorage()).personalKeys }); + personalKeyStoreContract({ create: async () => openStorage() }); personStoreContract({ create: async () => (await openStorage()).persons }); subscriberFailureStoreContract({ create: async () => (await openStorage()).subscriberFailures }); tableContract({ @@ -362,6 +362,7 @@ describe("sqlite adapter on a file", () => { ["checkpoints", 1], ["storage", 1], ["storage", 2], + ["storage", 3], ]); });