Distributed Webhook Delivery System

A high-performance, resilient Distributed Webhook Delivery System built with Node.js, TypeScript, BullMQ, Redis, PostgreSQL, and Express.js.

Includes exponential backoff retries, Dead Letter Queue (DLQ) support with manual replay capabilities, HMAC-SHA256 payload signing, idempotency headers, per-subscription rate limiting, circuit breaker pattern, and real-time monitoring via Bull Board.

---

Features

• Webhook Subscription CRUD API: Manage target URLs, event types, secrets, and active/inactive status.
• Event Ingestion Engine: Validates events, generates unique UUID event_id for idempotency, and dispatches jobs to BullMQ.
• Worker & Delivery Engine: Asynchronous job workers executing HTTP POST requests with 10s maximum timeout per attempt.
• HMAC-SHA256 Payload Signing: Signs every webhook with a secret key (X-Webhook-Signature: t=<timestamp>,v1=<hash>).
• Smart Retry Logic: Exponential backoff (1s, 2s, 4s, 8s, 16s) for 5xx errors & timeouts. Bypasses retries for 4xx client errors.
• Dead Letter Queue (DLQ): Failed/exhausted jobs are persisted to PostgreSQL dead_letter_queue table and BullMQ DLQ queue, with a REST API to inspect and replay entries.
• Rate Limiting: Sliding window Redis rate limiter per subscription to prevent worker starvation.
• Circuit Breaker: Redis-backed circuit breaker automatically trips after 10 consecutive failures and enforces a 60-second cooldown period.
• Delivery Logging: Comprehensive Postgres logging of every attempt (status, latency, status code, error message).
• Bull Board Dashboard: Real-time visualization of queue metrics at /admin/queues.

---

Tech Stack

• Node.js & TypeScript
• BullMQ: Queue management & job processing
• Redis: Queue backend, rate limiting, and circuit breaker state
• PostgreSQL: Subscriptions, delivery logs, and DLQ storage
• Express.js: REST API layer
• @bull-board/express: Web UI for monitoring queue jobs

---

Architecture Overview

+------------------+         +------------------+         +------------------+
|  Internal System |  POST   |  Express API     |  Enqueue|  BullMQ Queue    |
|  (Publisher)     | ------> |  /api/v1/events  | ------> |  (Redis)         |
+------------------+         +------------------+         +------------------+
                                                                  |
                                                                  v
                                                          +------------------+
                                                          | Delivery Worker  |
                                                          +------------------+
                                                            |     |     |
                                          +-----------------+     |     +-----------------+
                                          |                       |                       |
                                          v                       v                       v
                                   +--------------+       +---------------+       +---------------+
                                   | Circuit      |       | Webhook       |       | Delivery Logs |
                                   | Breaker /    |       | Subscriber    |       | & DLQ         |
                                   | Rate Limiter |       | Target URL    |       | (PostgreSQL)  |
                                   +--------------+       +---------------+       +---------------+

---

Quickstart Guide

1. Prerequisites

• Docker & Docker Compose
• Node.js (v18+) & npm

2. Start PostgreSQL & Redis Services

docker compose up -d

3. Install Dependencies

npm install

4. Start Development Server

npm run dev

The server will automatically initialize PostgreSQL tables and start on http://localhost:3000.

5. Access Queue Dashboard

Open http://localhost:3000/admin/queues in your browser to inspect queue activity in real-time.

---

Testing & Verification

Start Mock Receiver (Terminal 2)

npm run mock-receiver

Runs a mock receiver server on http://localhost:4000 with endpoints to simulate 200 success, 500 retries, 400 client failures, and timeouts.

Run Automated E2E Test Suite (Terminal 3)

npm run test:e2e

This script automatically:

1. Creates subscriptions for success and failure mock targets.
2. Ingests webhook events.
3. Verifies HMAC-SHA256 signature verification and HTTP headers.
4. Checks delivery logs in PostgreSQL.
5. Verifies DLQ entry creation for 400 non-retryable errors.
6. Replays DLQ entry and verifies successful redelivery.

---

REST API Reference

1. Webhook Subscriptions

Create Subscription

POST /api/v1/subscriptions

{
  "target_url": "https://example.com/webhook",
  "event_types": ["user.signup", "order.created"],
  "secret": "my-secret-key-123",
  "rate_limit": 100
}

List Subscriptions

GET /api/v1/subscriptions?event_type=user.signup

Get Subscription

GET /api/v1/subscriptions/:id

Update Subscription

PUT /api/v1/subscriptions/:id

{
  "is_active": false
}

Delete Subscription

DELETE /api/v1/subscriptions/:id

---

2. Event Ingestion

Ingest Internal Event

POST /api/v1/events

{
  "event_type": "user.signup",
  "payload": {
    "user_id": "usr_998877",
    "email": "user@example.com"
  }
}

---

3. Dead Letter Queue (DLQ)

List DLQ Entries

GET /api/v1/dlq?limit=50&offset=0

Get DLQ Entry Detail

GET /api/v1/dlq/:id

Replay DLQ Entry

POST /api/v1/dlq/:id/replay

---

4. Delivery Logs

Query Delivery Logs

GET /api/v1/deliveries?status=SUCCESS&limit=20

---

Webhook Headers Sent to Subscribers

Every webhook HTTP request sent to a subscriber target URL contains:

Header Name	Description	Example
Content-Type	MIME type	application/json
X-Webhook-ID	Unique Event UUID for Idempotency	e9b8a1c2-3d4e-5f6a-7b8c-9d0e1f2a3b4c
X-Webhook-Event	Subscribed event type	user.signup
X-Webhook-Timestamp	Unix timestamp of dispatch	1753900000
X-Webhook-Signature	HMAC-SHA256 signature	t=1753900000,v1=a8f9c7...