Stop jobs from failing when the stack has been idle, by keeping task results in Redis

[mihow]

Summary

Two kinds of job failure come from the same setting. Starting a job on a stack that has been idle for a minute or two can return a server error. Long synchronous processing jobs can end as Failed even though every batch was saved. Both happen because the local, CI and example production settings pin Celery's result backend to rpc://, which keeps task results on long-lived RabbitMQ connections. RabbitMQ closes those connections when they are idle past the heartbeat window, and the next status check runs into the closed connection.

This change drops that pin. The settings already fall back to Redis database 1 when the variable is unset, which is what the deployment described in #1189 uses. It also stops one high-volume task from copying its whole ML payload into the result backend on every batch.

This is the configuration-level fix for the two bugs that #1437 and #1443 fix in code. Those two PRs are still worth merging, and the reasons are below.

List of Changes

Change (effect)	How	Notes
1. Starting a job and finishing a synchronous ML job no longer depend on a RabbitMQ connection staying alive while idle.	Remove CELERY_RESULT_BACKEND=rpc:// from .envs/.local/.django, .envs/.ci/.django and .envs/.production/.django-example. config/settings/base.py then derives redis://…/1 from REDIS_URL.	Deployments whose own env file still sets rpc:// need the same line removed.
2. Each batch of results from pull-mode processing services is no longer stored a second time in the result backend.	ignore_result=True on process_nats_pipeline_result.	The batch is still validated and saved to the database, then acknowledged. Only Celery's leftover result record is skipped.
3. A test pins that this task stores no result.	TestProcessNatsPipelineResultStoresNoResult.

Detailed Description

What was measured

On an isolated stack built from main, with no mocks:

	With rpc://	With Redis DB 1
Create and start a job after about 100 s idle	HTTP 500, ConnectionResetError: [Errno 104] from Job.enqueue() reading AsyncResult(task_id).status (3 of 3)	201 every time
Synchronous ML job, with the worker idle between jobs	FAILURE with processing at 100% and every save_results sub-task succeeded. The errors were [Errno 104] from wait(), then TimeoutError on the next job (2 of 2).	SUCCESS on every job, 27 wait() calls without an error, including after 150 s idle
ami.jobs and ami.ml tests		302 passed, 2 skipped

RabbitMQ logged missed heartbeats from client, timeout: 30s for the dropped connections. Under Redis, RabbitMQ still closes idle connections, but the only remaining operation on them is publishing a task, which kombu retries. The status reads that failed now go to Redis.

Why process_nats_pipeline_result in particular

Celery task state is read in these places:

• run_job: in Job.enqueue(), in the stale-job sweep, and in Job.update_status() when no status is passed.
• The save_results sub-tasks: in the synchronous ML job's wait.

Those keep their stored results. Nothing reads the state of process_nats_pipeline_result. The processing service receives its task id in the HTTP response but does not use it. Because CELERY_RESULT_EXTENDED = True stores task arguments with each result, every stored result held a copy of the full batch payload. #1189 measured an average of 191 KB per key, with thousands of keys per async job kept for 72 hours.

A side benefit: with rpc://, a task's state cannot be read from another process, so it always reads as PENDING. The stale-job sweep therefore never saw run_job finish. With Redis it sees the real state.

Relationship to #1437 and #1443

Both remain useful with Redis as the result backend.

• Stop job creation from failing when the result backend connection is idle #1437 stops Job.enqueue() from asking the backend for the status of a task it has just sent. That status is always pending. The PR also removes a backend round trip from inside the request transaction.
• Stop synchronous processing jobs from failing after their results are saved #1443 replaces wait() inside a task with polling ready() and a stall limit. Celery documents calling wait() inside a task as deadlock-prone, and the old 60-second limit per batch could fail a job whose last save legitimately takes longer.

How to test

• python manage.py test ami.jobs
• On a running stack with this branch: leave it idle for two minutes, then start a job from the UI. Run a synchronous ML job, wait two minutes, and run another. Both should succeed. redis-cli -n 1 --scan | head shows result keys for run_job and save_results, but none for process_nats_pipeline_result.

Open points

• If Redis is shared with the cache under an allkeys-lru eviction policy, an evicted result reads as PENDING. This is inferred, not measured. Separate instances or eviction policies are discussed in Evaluate Celery result backend and broker architecture #1189.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C7Xf6VPbwWtTumhjjF15g8

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	73466a1
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6abef05fe6d0a90007caf545

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[netlify]

✅ Deploy Preview for antenna-ssec canceled.

Name	Link
🔨 Latest commit	73466a1
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-ssec/deploys/6abef05fce8c5e0008ed31de