Fix dangling processing jobs by adding TCP keepalives #1072

Summary

Celery workers may be creating "Zombie Tasks" due to a 1-hour RabbitMQ connection hard close caused by the OpenStack network's idle timeout. The existing Linux default TCP Keepalive is too slow, leading to failed task completion reports.

(issue description written with the help of Gemini based on the problem analysis)

Detailed Description

Our Celery implementation is experiencing two distinct connection issues, the most severe of which is the 1-hour "Hard Close".

1. The "Hard Close" (Zombie Task Maker):
   • Cause: The OpenStack networking layer (Neutron) has an Infrastructure Idle Timeout of exactly 1 hour (3600 seconds).
   • Mechanism: When long-running "Master Tasks" wait silently for subtasks or perform local work for over an hour, the connection is considered "idle" and silently severed by the firewall. When the task eventually tries to report "Success," the network rejects the communication, crashing the worker and leaving the task permanently stuck in a "Started" state in the database.
   • Symptom: Logs show client unexpectedly closed TCP connection or ConnectionResetError: [Errno 104].
2. The "Soft Close" (CPU Blocker):
   • Cause: Thread Starvation in the worker process (Python's GIL) during heavy CPU tasks.
   • Mechanism: The worker cannot send its application-level heartbeat (CELERY_BROKER_HEARTBEAT = 30). RabbitMQ closes the connection after 3 missed heartbeats (90 seconds).
   • Symptom: Logs show missed heartbeats from client, timeout: 30s.

Importance: The Zombie Task issue is critical as it corrupts task status tracking in the database and prevents long-running jobs from accurately reporting completion.

Possible Fix Implementation

The primary fix is to configure OS-Level TCP Keepalives to probe the connection every 60 seconds, which is less than the 1-hour firewall timeout.

• Specific Files/Areas: settings.py (for Celery configuration).
• Approximate timeline: Immediate deployment (after testing) as it is a critical connection stability fix.

Implementation Tasks:

1. Update settings.py by replacing the existing CELERY_BROKER_TRANSPORT_OPTIONS block with the following, which explicitly sets the TCP keepalive parameters and removes the self-destructing socket_timeout.

import socket  # Required at the top of your settings.py

CELERY_BROKER_TRANSPORT_OPTIONS = {
    # 1. THE CRITICAL FIX: Custom TCP Keepalives
    "socket_keepalive": True,
    "socket_settings": {
        # Start sending Keepalive packets after 60 seconds of silence.
        # This forces traffic on the wire, preventing the OpenStack 1-hour timeout.
        socket.TCP_KEEPIDLE: 60,
          
        # If no response, retry every 10 seconds.
        socket.TCP_KEEPINTVL: 10,
          
        # Give up and close connection after 9 failed attempts.
        socket.TCP_KEEPCNT: 9,
    },
  
    # 2. Connection Stability Settings
    "socket_connect_timeout": 40,  # Max time to establish connection
    "retry_on_timeout": True,      # Retry operations if they time out
    "max_connections": 20,         # Connection pool limit per process
  
    # 3. SAFETY REMOVAL: "socket_timeout"
    # We REMOVED "socket_timeout: 120" to prevent workers self-destructing during long blocking operations.
}
  
# 4. Application Heartbeat (Keep this active)
CELERY_BROKER_HEARTBEAT = 30

2. Verify the Configuration after deploying changes and restarting workers (see Additional context below).

Future Improvement (Architectural Refactoring)

For multi-hour jobs composed of a single long master task and many fast subtasks, refactor them into Celery Chords. This will allow the Master task to shut down immediately after dispatch, eliminating the idle connection issue entirely and improving resource utilization.

How to Reproduce the Bug

Environment (please complete the following information)

• OS and/or device: Worker VM on Compute Canada (Arbutus OpenStack)
• Browser: [N/A]
• Version: [e.g. Production, Development, etc.]

Steps to reproduce the behavior:

1. Start a "Master Task" (Processing Job) that is designed to be silent (no network I/O) for a period longer than 1 hour (3600 seconds) while it waits for subtasks or performs local calculations.
2. After the 1-hour mark, the OpenStack firewall silently severs the connection to the RabbitMQ broker.
3. When the Master Task finishes its work and attempts to report its SUCCESS status back to the broker/backend, the network operation fails.
4. See the worker crash with ConnectionResetError and observe the task status in the database permanently stuck at "Started" (Zombie Task).

Additional context

To verify the fix is working on the OS level, SSH into a Worker VM after deployment and run the connection timer check.

# This lists TCP connections to port 5672 with timer details
sudo ss -tonp | grep 5672

Expected Output: Look for a value counting down from ~60 seconds in the timer column: timer:(keepalive,59sec,0). If you see a value like 7199sec or 'off', the new settings were not applied correctly.