Make AaaS delivery durable and restart-safe

[coodos]

What was rotting

• The delivery loop used a permanent in-memory running latch. One unresolved database call could leave the latch set forever, so every later poll was skipped while the health endpoint still returned 200.
• eVault committed user data before making a short-lived fire-and-forget HTTP call to AaaS. An AaaS or network outage after the data commit permanently lost the awareness event.
• Event persistence and delivery creation were not one atomic operation, and content-based deduplication incorrectly collapsed legitimate A to B to A changes.

Fix

• Add a transactional Neo4j outbox to every eVault mutation and retry ingestion until AaaS durably acknowledges it.
• Store immutable awareness events and matching deliveries atomically in Postgres with stable event IDs and monotonic stream versions.
• Replace the wedging poll loop with self-scheduling, deadline-bounded delivery batches, expiring token-fenced leases, crash recovery, per-stream ordering, and a 24-hour retry window.
• Add real readiness and Prometheus metrics for queue age, leases, database state, and worker heartbeat.
• Repair catch-all reconciliation and backfill behavior.
• Keep deployment simple: the HTTP API and delivery worker run together inside one AaaS process. No second worker service or process is required.

Verification

• AaaS TypeScript build passes.
• AaaS durability/recovery tests: 4 passed.
• eVault awareness and database tests: 27 passed across the targeted suites; the complete eVault run accounted for 250 tests.
• Documentation production build passes (existing broken-anchor warnings remain).
• Git diff whitespace checks pass.

Deployment required

This changes both AaaS and eVault and adds a Postgres migration. After merge:

1. Build AaaS and run the AaaS migration before startup.
2. Deploy one AaaS process; it runs both the API and worker internally.
3. Build and redeploy eVault so mutations use the durable outbox.
4. Verify AaaS /ready and eVault /ready, then monitor /metrics.

Do not create separate API and worker deployments.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 2392fd50-b437-435b-8842-3c696a3e89c6

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}