Review how the state and progress of jobs are tracked #1285

Summary

The platform tracks an async_api job's lifecycle in three independent
places — Job model in Postgres, AsyncJobStateManager in Redis, and
JetStream consumer state in NATS. Each has different failure modes, and
none alone catches every observed bug class. We should expose a unified
diagnostic view that triangulates all three.

This is a low-priority follow-up to #1276, which patched the immediate
"reaper REVOKES on clobbered Job.progress" loop by reading Redis
directly. Triangulation is the next layer up — forensic visibility and
catching adjacent bug classes that Redis alone can't see.

Why three sources of truth lie differently

DB alone catches none reliably. Redis catches most. Redis+NATS catches all.

Proposal

Three pieces, mostly independent:

1. Snapshot NATS state on terminal transition. On SUCCESS / FAILURE /
   REVOKED, persist consumer counters into Job.progress.diagnostics (or
   a new Job.diagnostics JSONB). Cheap. NATS consumer state is deleted
   post-completion — without a snapshot the forensic trail is gone.

2. GET /api/v2/jobs/<id>/diagnostics endpoint returning structured
   triangulation:

   • db — Job.status, Celery state, progress.is_complete(), per-stage progress
   • redis — per-stage pending/total, failed set, all_tasks_processed, TTL
   • nats — stream/consumer, delivered, ack_floor, num_pending, num_ack_pending, num_redelivered
   • divergence — server-side computed flags (DB-vs-Redis, DB-vs-NATS, Redis-vs-NATS mismatches with severity)

   Cache 5–10s. NATS info is an async socket round-trip; cheap but not free.

3. JobReconciler.diagnose() abstraction. Promote the tri-state
   AsyncJobStateManager.all_tasks_processed() added in fix(jobs): fix dangling jobs from going to revoked #1276 to a
   richer diagnose() returning the full triangulation. Both the
   diagnostics endpoint and the reaper guard call the same function.
   Single source of triangulation logic.

4. UI diagnostic panel (admin / power-user). Keep default progress
   bars unchanged. Add expandable section:

   process     dispatched  acked   pending  failed  redelivered
               450/450     445     5        0       12
               ── NATS ──  ─NATS─  ─Redis─  Redis   NATS
   

   Plus header strip with consumer name, Redis TTL, divergence flags,
   and a "Force reconcile" admin button that re-runs the reaper guard
   on demand.

Tradeoffs

• NATS info call needs JetStream credentials reachable from Django (already
  have these via the natsconn integration).
• More numbers = more confusion for non-admins — gate the diag panel by
  permission.
• NATS consumer state goes stale fast post-completion (consumers deleted).
  Snapshot-on-terminal mitigates this.
• Adds a per-job NATS round-trip on endpoint hit. Cache + admin-only
  surface keeps cost bounded.

Recommended order

1. Snapshot-on-terminal first — cheap, no UI, big forensic value.
2. Diagnostics endpoint + JSON shape. Lets reaper + admin tooling consume
   it; no UI needed yet.
3. Promote reaper to consume JobReconciler.diagnose() — replaces inline
   tri-state branch from fix(jobs): fix dangling jobs from going to revoked #1276.
4. UI panel last, once endpoint shape is stable.

Related issues

• Follow-ups from PR #1231 review: remaining Redis/retry gaps in async job state #1232 — Redis-only follow-ups from PR fix: retry on Redis blips instead of failing the job #1231
• Ensure that a processing job's status is always updated correctly #721 — backend bug: status stuck STARTED (user-facing symptom this would help debug)
• PSv2: Integrate with the incomplete job monitoring #1085 — PSv2: integrate incomplete job monitoring
• Add COMPLETED job state for jobs that finish with partial errors #1134 — COMPLETED job state for partial-error jobs
• feat: PSv2 - Have visibility into which workers are subscribed to a job #1112 — Worker visibility per job
• Improve visibility of processing service status in UI #1166 — Processing service status indicator (adjacent UI work)
• PSv2: Async jobs hang forever when NATS tasks exhaust max_deliver without posting results #1168 — Async jobs hang when NATS max_deliver exhausts
• A few images are regularly stranded after long-running jobs #1247 — Lost-image stranding (a bug class triangulation would surface)
• fix(jobs): fix dangling jobs from going to revoked #1276 — Reaper Redis check (the narrower fix this builds on)

Reference

Full breakdown captured in docs/claude/planning/job-state-triangulation.md
in the same PR series.