`dealbot-worker` pods on Infra Staging (calibration) have been crash-looping for at least 9 days. Pod `dealbot-worker-...-qwdjd` alone shows 39 restarts since 2026-09-09; all 4 replicas show the same pattern. 76 of 77 unhandled-rejection crashes fleet-wide in the last 7 days share one cause.
What happens
pg-boss's internal maintenance timer (`Timekeeper.onCron` / `Manager.onCacheQueues`) occasionally fails to acquire a Postgres connection and throws unguarded:
Error: timeout exceeded when trying to connect
at pg-pool/index.js:45:11
at async Db.executeSql (pg-boss/dist/db.js:43:16)
at async Manager.getQueues (pg-boss/dist/manager.js:645:26)
at async Manager.onCacheQueues (pg-boss/dist/manager.js:210:28)
This surfaces as a Node unhandled promise rejection (`level:"fatal", event:"unhandled_rejection"`) that nothing catches, crashing the process. Kubernetes then restarts the container.
Contributing factor
`DEALBOT_PGBOSS_POOL_MAX` defaults to `1` (`apps/backend/src/config/env.schema.ts:106`) — a single Postgres connection shared by pg-boss's 5 job queues plus its own internal timers in one process, easy to starve under any DB latency.
Impact
Each crash silently drops whatever was in flight. Most visible on low-frequency global jobs with no retry: `providers_refresh` (4h interval) has had recurring 8-32h gaps with zero completions (success or error) on calibration over the past week. High-volume per-SP jobs (deal/retrieval/etc.) mask the crashes since the other replicas keep serving those queues.
Not currently alerted — existing BetterStack job-health alerts on the Dealbot Operational Dashboard are scoped to `network='mainnet'` only.
Environment: calibration / Infra Staging, dealbot-backend v1.13.1, pg-boss 12.11.1, pg-pool 3.11.0
`dealbot-worker` pods on Infra Staging (calibration) have been crash-looping for at least 9 days. Pod `dealbot-worker-...-qwdjd` alone shows 39 restarts since 2026-09-09; all 4 replicas show the same pattern. 76 of 77 unhandled-rejection crashes fleet-wide in the last 7 days share one cause.
What happens
pg-boss's internal maintenance timer (`Timekeeper.onCron` / `Manager.onCacheQueues`) occasionally fails to acquire a Postgres connection and throws unguarded:
This surfaces as a Node unhandled promise rejection (`level:"fatal", event:"unhandled_rejection"`) that nothing catches, crashing the process. Kubernetes then restarts the container.
Contributing factor
`DEALBOT_PGBOSS_POOL_MAX` defaults to `1` (`apps/backend/src/config/env.schema.ts:106`) — a single Postgres connection shared by pg-boss's 5 job queues plus its own internal timers in one process, easy to starve under any DB latency.
Impact
Each crash silently drops whatever was in flight. Most visible on low-frequency global jobs with no retry: `providers_refresh` (4h interval) has had recurring 8-32h gaps with zero completions (success or error) on calibration over the past week. High-volume per-SP jobs (deal/retrieval/etc.) mask the crashes since the other replicas keep serving those queues.
Not currently alerted — existing BetterStack job-health alerts on the Dealbot Operational Dashboard are scoped to `network='mainnet'` only.
Environment: calibration / Infra Staging, dealbot-backend v1.13.1, pg-boss 12.11.1, pg-pool 3.11.0