[pournasserian]

Closes #2618

What

The recurring retention purge of stored prompts (AUD-04), from the post-merge review of #2545: until now nothing purged them unless a person called IModelCallLog.PurgeAsync, and SystemInitiator.Retention was unused.

• retention.purge job (RetentionPurgeJob, Core):
  • It runs as SystemInitiator.Retention and is listed for it in EngineJobs (Harden the system initiator: reset links, an engine allow-list and refusal tests #2617).
  • It runs only as the retention engine; a run as anyone else fails and purges nothing.
  • It first schedules its next run a day later, so a run that fails or dies after that still leaves one coming. A run whose every attempt fails before its scheduling commits leaves none until a host restarts; Keep the retention purge scheduled when a run dies before scheduling its successor #2652 follows that up.
  • It then calls PurgeAsync batch after batch until a batch clears nothing, each batch in a scope of its own so a long backlog does not pile up in one context, reporting progress after each.
  • Each cleared call keeps its model_call.purged event, logged as the retention engine. A purge the engine may not make fails the run with its reason.
• A run is always scheduled (IRetentionSchedule):
  • When a host starts, after the database is migrated, it enqueues a run as the retention engine if none is queued (RetentionStartup). A database the host cannot reach yet only delays that to the next start.
  • A run enqueues its successor before it purges, unless another run is queued. So a running run always leaves one queued, and the schedule is never empty.
• PurgeAsync's comment says it clears a batch at a time.
• CONTRIBUTING.md: the retention purge.

Why

AUD-04 sets a retention for stored prompts and responses. A retention that a person must remember to apply is not enforced, so the product applies it itself, as the engine ADR-0012 named for it.

Choices worth a look:

• Daily, not configurable. It is well inside any retention of days, and a setting can come when someone needs one.
• Scheduled from the job and from startup, rather than a timer in the host, so the schedule lives in the job queue (ADR-0010) and survives restarts, and any host can run the purge.
• Enqueued as the retention engine through a scope only Core opens, so the job records the engine as the one that asked for it.
• Never empty rather than exactly one. Without a unique slot in the job queue, two runs that start together can each see the other and must not both skip, or the schedule ends. Both enqueue, the purge is safe to repeat, and the next day's runs fold the two chains back into one.
• Builds on Harden the system initiator: reset links, an engine allow-list and refusal tests #2617 for EngineJobs, now in v2.

Tests

Integration, on SQLite and PostgreSQL where a database is involved:

• A started host's worker runs the purge as the retention engine. The expired call is cleared, model_call.purged names the engine, and one run is left queued a day later. A second host that starts then enqueues none. The test waits for the run to succeed, and the first host's own run ends before the expired call goes in.
• A run purges 1,101 calls past the retention across two batches, reporting 1,000 then 1,101. It logs model_call.purged for each as the retention engine, and schedules the next run a day later as that engine.
• A run as the retention engine that finds one already queued schedules none.
• A purge that fails fails the run with its reason, and the next run is still scheduled.
• A run made as another engine fails with model_purge_people_only, and purges and schedules nothing.
• Two runs at once leave at least one run queued.
• EngineJobs lists the purge for the retention engine (unit).

The full suite ran locally against a postgres:17 container as well as SQLite, and markdownlint passes. No new package and no migration. After v2's webhook job, the model test scenario gives the job handlers a secret protector stub, since the retention tests resolve them together as the worker does. A started host now runs a retention job at once, so a test that reads the SQLite file while its host runs, as the webhook tests do, reads it without asking for sole access.

Checklist

• Linked to an issue a maintainer has confirmed
• Targets v2
• Tests or docs updated where behaviour changed