MetaState-Prototype-Project · coodos · Sep 14, 2026 · Sep 14, 2026 · Sep 14, 2026
@@ -154,9 +154,23 @@ DO_SPACES_BUCKET="your-spaces-bucket"
 DO_SPACES_CDN_URL=""
 # Secret used to sign AaaS portal session JWTs
 AAAS_JWT_SECRET="replace-with-a-strong-secret"
-# Webhook delivery tuning
-AWARENESS_MAX_ATTEMPTS=3
+# Webhook delivery worker runs in the same AaaS process as the HTTP API.
 AWARENESS_DELIVERY_POLL_MS=2000
+AWARENESS_DELIVERY_LEASE_MS=30000
+AWARENESS_DELIVERY_BATCH_TIMEOUT_MS=25000
+# Retry downstream webhooks for 24 hours before dead-lettering.
+AWARENESS_DELIVERY_RETRY_WINDOW_MS=86400000
+AWARENESS_WORKER_HEARTBEAT_MS=10000
+AWARENESS_WORKER_STALE_MS=30000
+# Bound every Postgres operation so one poisoned connection cannot wedge AaaS.
+AWARENESS_DB_STATEMENT_TIMEOUT_MS=10000
+AWARENESS_DB_QUERY_TIMEOUT_MS=12000
+AWARENESS_DB_LOCK_TIMEOUT_MS=5000
+# Durable eVault -> AaaS outbox tuning. Ingest retries never expire.
+AWARENESS_OUTBOX_POLL_MS=1000
+AWARENESS_OUTBOX_LEASE_MS=30000
+AWARENESS_OUTBOX_DB_TIMEOUT_MS=10000
+AWARENESS_OUTBOX_RETENTION_MS=604800000
 # The one-time Neo4j backfill reuses the standard NEO4J_URI / NEO4J_USER /
 # NEO4J_PASSWORD vars at the top of this file - it reads evault-core's graph
 # directly, so there are no AaaS-specific Neo4j vars.

@@ -0,0 +1,63 @@
+name: Tests [awareness-service]
+
+on:
+    push:
+        branches: [main]
+    pull_request:
+        branches: [main]
+        paths:
+            - "services/awareness-service/**"
+            - "infrastructure/evault-core/src/core/awareness/**"
+            - "infrastructure/evault-core/src/core/db/**"
+            - "infrastructure/evault-core/src/core/protocol/**"
+            - "pnpm-lock.yaml"
+
+jobs:
+    test:
+        runs-on: ubuntu-latest
+
+        steps:
+            - name: Checkout code
+              uses: actions/checkout@v4
+
+            - name: Set up pnpm
+              uses: pnpm/action-setup@v4
+              with:
+                  version: 10.25.0
+                  run_install: false
+
+            - name: Set up Node.js 22
+              uses: actions/setup-node@v4
+              with:
+                  node-version: 22
+                  cache: pnpm
+
+            - name: Install build dependencies
+              run: |
+                  sudo apt-get update
+                  sudo apt-get install -y build-essential python3
+
+            - name: Install dependencies
+              run: pnpm install --frozen-lockfile
+
+            - name: Build affected services
+              run: |
+                  pnpm --filter awareness-service-api build
+                  pnpm --filter evault-core build
+
+            - name: Run AaaS reliability tests
+              env:
+                  CI: true
+                  DOCKER_HOST: unix:///var/run/docker.sock
+                  TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE: /var/run/docker.sock
+                  TESTCONTAINERS_HOST_OVERRIDE: localhost
+              run: pnpm --filter awareness-service-api test
+
+            - name: Run eVault awareness outbox tests
+              env:
+                  CI: true
+                  DOCKER_HOST: unix:///var/run/docker.sock
+                  TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE: /var/run/docker.sock
+                  TESTCONTAINERS_HOST_OVERRIDE: localhost
+              run: |
+                  pnpm --filter evault-core exec vitest run src/core/protocol/graphql-server.spec.ts src/core/protocol/uploadFile-awareness.spec.ts --silent
@@ -649,20 +649,22 @@ A valid Registry-issued platform token satisfies the *legacy* path — but it do
 
 ## Webhook Delivery
 
-When data is stored or updated, eVault automatically sends webhooks to all registered platforms.
+When data is created, updated, or deleted, eVault atomically records an
+awareness outbox event beside the mutation. A restart-safe dispatcher sends it
+to AaaS, which owns subscription matching and webhook delivery.
 
 ### Webhook Process
 
-1. **Data Stored**: MetaEnvelope is stored in Neo4j
-2. **Wait 3 Seconds**: Delay prevents webhook ping-pong (same platform receiving its own webhook)
-3. **Get Active Platforms**: Query [Registry](/docs/Infrastructure/Registry) for list of active platforms
-4. **Filter Requesting Platform**: Exclude the platform that made the request
-5. **Send Webhooks**: POST to each platform's `/api/webhook` endpoint (see [Webhook Controller Guide](/docs/Post%20Platform%20Guide/webhook-controller))
+1. **Atomic capture**: The MetaEnvelope mutation and immutable outbox event commit in one Neo4j transaction.
+2. **Durable ingest**: The outbox dispatcher retries `POST AWARENESS_SERVICE_URL/ingest` until AaaS acknowledges persistence.
+3. **Match and filter**: AaaS matches subscriptions and excludes the requesting platform.
+4. **Send webhooks**: AaaS posts to each matching endpoint (see [Webhook Controller Guide](/docs/Post%20Platform%20Guide/webhook-controller)).
 
 ### Webhook Payload
 
 ```json
 {
+    "eventId": "7fd6c06c-80ae-4137-9d62-c15af53f92cf",
     "id": "global-id-123",
     "w3id": "@user-a.w3id",
     "schemaId": "550e8400-e29b-41d4-a716-446655440001",
@@ -672,16 +674,19 @@ When data is stored or updated, eVault automatically sends webhooks to all regis
         "authorId": "...",
         "createdAt": "2025-01-24T10:00:00Z"
     },
-    "evaultPublicKey": "z..."
+    "evaultPublicKey": "z...",
+    "operation": "update",
+    "streamVersion": 2,
+    "occurredAt": "2026-09-15T03:00:00.000Z"
 }
 ```
 
 ### Webhook Delivery Details
 
-- **Timeout**: 5 seconds per webhook
-- **Retry**: No automatic retries (fire-and-forget)
-- **Error Handling**: Logs failures but doesn't block the operation
-- **Ordering**: Webhooks are sent in parallel to all platforms - sending to platform A does not block sending to platform B. All webhook POST requests are initiated concurrently.
+- **Timeout**: 5 seconds per network attempt.
+- **Retry**: eVault-to-AaaS retries until acknowledged; AaaS-to-subscriber retries with backoff for 24 hours and then dead-letters.
+- **Idempotency**: Subscribers must deduplicate by `eventId` because delivery is at least once.
+- **Ordering**: Events are ordered per subscription and MetaEnvelope while unrelated streams are delivered concurrently.
 
 ## Key Binding Certificates
 

@@ -22,16 +22,17 @@ to all of them. That design had three problems:
 - **Ungoverned** — any registered platform received everything; there was no
   access gate.
 
-AaaS fixes all three. evault-core now makes **one** POST per change to
-`AWARENESS_SERVICE_URL/ingest`, and AaaS owns persistence, polling, subscription
-matching, and retrying delivery.
+AaaS fixes all three. Every eVault mutation now commits an immutable event to a
+Neo4j transactional outbox alongside the user's data. The outbox retries
+`AWARENESS_SERVICE_URL/ingest` until AaaS atomically commits the event and its
+matching deliveries; AaaS then owns polling and retrying subscriber delivery.
 
 ## Architecture
 
 ```
                          ┌─────────────────────────────┐
-   evault-core ──POST───▶ │  AaaS  /ingest              │
-   (per change)           │   • persist packet          │
+   eVault outbox ─POST───▶ │  AaaS  /ingest              │
+   (retry-until-ack)       │   • persist immutable event │
                           │   • match subscriptions     │
                           │   • queue deliveries        │
                           └──────────────┬──────────────┘
@@ -63,9 +64,14 @@ existing receivers need no changes:
 }
 ```
 
-`/ingest` additionally accepts a `requestingPlatform` field, used only to skip
+New producers add `eventId`, `streamVersion`, and `occurredAt`. `eventId` is the
+stable idempotency key across outbox retries, while `id` remains the
+MetaEnvelope id. These fields are delivered additively to existing receivers.
+
+`/ingest` additionally accepts a `requestingPlatform` field, used to skip
 delivering a packet back to its origin (the ping-pong guard the old fanout
-enforced). It is never persisted or delivered.
+enforced). It is retained in immutable event history for audit/reconciliation,
+but is not included in subscriber payloads.
 
 ### File uploads
 
@@ -113,16 +119,16 @@ subscription has a `secret`, each delivery carries an `x-aaas-signature` header
 (HMAC-SHA256 of the body).
 
 Because catch-all subscriptions receive every ontology, a receiver **must ack
-packets it does not consume with a 200**. There is no 4xx short-circuit in the
-delivery engine: a 400 on an unknown `schemaId` is retried up to
-`AWARENESS_MAX_ATTEMPTS` and then dead-lettered.
+packets it does not consume with a 200**. All non-2xx responses remain retryable
+for the 24-hour window, after which the event is dead-lettered and alerted.
 
 ### 3. Retrying delivery + dead-letters
 
-A background engine drains the delivery queue. Failed deliveries are retried
-with exponential backoff (30s → 1m → 2m → 5m → 15m → 1h → 6h → 24h). After
-`AWARENESS_MAX_ATTEMPTS` attempts the delivery is moved to a **dead-letter**
-table, visible to admins in the portal, where it can be replayed.
+A lease-based worker drains the delivery queue. Every Postgres operation and
+batch has a deadline, so a poisoned connection cannot permanently wedge the
+polling loop. Failed deliveries use jittered exponential backoff for 24 hours;
+expired leases are reclaimed after crashes. After the retry window the delivery
+moves to a **dead-letter** table, visible to admins in the portal for replay.
 
 ### 4. Public access portal
 
@@ -157,14 +163,16 @@ AaaS is designed to be dropped in with **zero receiver-side changes**:
 
 1. **Backfill.** AaaS runs on the same node as evault-core's Neo4j. The
    `backfill` script reads existing MetaEnvelopes straight from the graph and
-   seeds the `packets` table (history only — it does not queue deliveries).
+   seeds both immutable query history and the latest-state projection. It does
+   not queue deliveries.
 2. **Catch-all reconciliation.** On every launch and once per configured sync
    interval, AaaS ensures each platform currently in the registry has an
    approved consumer and an active catch-all subscription pointing at
    `<platform>/api/webhook`. Existing and newly registered platforms therefore
    keep receiving every packet exactly as before.
-3. **evault-core switch.** evault-core's `deliverWebhooks`/`getActivePlatforms`
-   are removed; a single `notifyAwareness` POST forwards each packet to AaaS.
+3. **eVault transactional outbox.** Every mutation and its awareness event
+   commit together in Neo4j. A dispatcher retries ingestion until AaaS returns a
+   durable acknowledgement, including across eVault and AaaS restarts.
 
 ## Configuration
 
@@ -177,8 +185,12 @@ AaaS is designed to be dropped in with **zero receiver-side changes**:
 | `AWARENESS_SERVICE_URL` | (evault-core) where to POST packets |
 | `AAAS_ADMIN_ENAMES` | Comma-separated admin eNames |
 | `AAAS_JWT_SECRET` | Signs portal session JWTs |
-| `AWARENESS_MAX_ATTEMPTS` | Delivery attempts before dead-lettering (default 3) |
 | `AWARENESS_DELIVERY_POLL_MS` | Delivery engine poll interval (default 2000) |
+| `AWARENESS_DELIVERY_LEASE_MS` | Expiring worker lease duration (default 30000) |
+| `AWARENESS_DELIVERY_BATCH_TIMEOUT_MS` | Hard batch deadline (default 25000) |
+| `AWARENESS_DELIVERY_RETRY_WINDOW_MS` | Subscriber retry window (default 24 hours) |
+| `AWARENESS_DB_STATEMENT_TIMEOUT_MS` / `AWARENESS_DB_QUERY_TIMEOUT_MS` / `AWARENESS_DB_LOCK_TIMEOUT_MS` | Postgres anti-wedge deadlines |
+| `AWARENESS_OUTBOX_POLL_MS` / `AWARENESS_OUTBOX_LEASE_MS` / `AWARENESS_OUTBOX_DB_TIMEOUT_MS` / `AWARENESS_OUTBOX_RETENTION_MS` | Durable eVault outbox tuning |
 | `AWARENESS_REGISTRY_SYNC_MS` | Registry catch-all reconciliation interval (default 60000; 0 disables periodic sync) |
 | `NEO4J_URI` / `NEO4J_USER` / `NEO4J_PASSWORD` | Standard eVault Neo4j vars — reused by the one-time backfill |
 | `PUBLIC_AWARENESS_API_URL` | (portal) AaaS API base URL |
@@ -190,6 +202,6 @@ AaaS is designed to be dropped in with **zero receiver-side changes**:
 pnpm --filter awareness-service-api build
 pnpm --filter awareness-service-api migration:run
 pnpm --filter awareness-service-api backfill        # one-time, from Neo4j
-pnpm --filter awareness-service-api dev             # API (keeps registry catch-alls synced)
+pnpm --filter awareness-service-api dev # API + worker in one process
 pnpm --filter awareness-portal dev                  # portal
 ```
@@ -78,14 +78,14 @@ Duplication is its own failure. [File URIs](/docs/W3DS%20Protocol/File-URIs) mak
 
 ## Bounds on how much a projection can be trusted
 
-Synchronisation is eventual, and the [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) is prototype-level. Design the projection to tolerate all of this:
+Synchronisation is eventual, and the [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) is delivered at least once. Design the projection to tolerate all of this:
 
 - **Last-write-wins.** No merge, no CRDT.
-- **No ordering guarantee**, and no at-least-once delivery.
-- **Fire-and-forget fanout** with no retries at the protocol level. The requesting platform is excluded from its own fanout.
-- **A delay after creation** before fanout, to prevent ping-pong; updates fan out immediately.
+- **Per-stream ordering, not global ordering.** Events for one subscription and MetaEnvelope are ordered; independent streams are concurrent.
+- **At-least-once delivery.** Retries and crash recovery can produce duplicates; the requesting platform is excluded from its own fanout.
+- **A bounded subscriber retry window.** AaaS retries for 24 hours and then requires dead-letter replay.
 
-Consequences for your code: webhook handling must be **idempotent on the global `id`**, reads must tolerate a record that has not arrived yet, and nothing user-visible should depend on two platforms agreeing at the same instant.
+Consequences for your code: webhook handling must deduplicate by **`eventId`** (not the MetaEnvelope `id`, which is shared by legitimate updates), reads must tolerate a record that has not arrived yet, and nothing user-visible should depend on two platforms agreeing at the same instant.
 
 ## Stateless applications
 

@@ -149,13 +149,16 @@ mutation CreateMetaEnvelope($input: MetaEnvelopeInput!) {
 
 The [eVault](/docs/Infrastructure/eVault) stores the data as a [MetaEnvelope](/docs/Infrastructure/eVault#data-model), which is a flat graph structure of Envelopes. Each field becomes a separate Envelope node in Neo4j.
 
-#### 6. Webhook Delivery (After 3 Second Delay)
+#### 6. Durable Webhook Delivery
 
-After a 3-second delay (to prevent webhook ping-pong), the [eVault](/docs/Infrastructure/eVault) sends webhooks to all registered platforms (see [Registry](/docs/Infrastructure/Registry)) **except** the one that made the request (Blabsy).
+The eVault commits an outbox event with the MetaEnvelope write. AaaS ingests it,
+then sends it to matching platforms **except** the one that made the request
+(Blabsy). Both handoffs are restart-safe and retry automatically.
 
 The webhook payload contains:
 ```json
 {
+    "eventId": "7fd6c06c-80ae-4137-9d62-c15af53f92cf",
     "id": "global-id-123",
     "w3id": "@user-a.w3id",
     "schemaId": "550e8400-e29b-41d4-a716-446655440001",
@@ -164,7 +167,10 @@ The webhook payload contains:
         "mediaUrls": [],
         "authorId": "...",
         "createdAt": "2025-01-24T10:00:00Z"
-    }
+    },
+    "operation": "create",
+    "streamVersion": 1,
+    "occurredAt": "2026-09-15T03:00:00.000Z"
 }
 ```