Feature: Slack Alerts

[rosemcc]

This pull request introduces a Slack notification system for archive submission and retrieval jobs, allowing alerts to be sent on job completion or failure. This is useful for providing visibility into the status of these background, potentially long-running tasks.

The Slack integration uses CeR's existing Slack app "Keith", a friendly but diseased "ratbot". Messages are sent via an incoming webhook configured on Keith.

Notification screenshot:

Notification system integration

• Added a new notify_job_result helper in src/service/notifications.py to send best-effort Slack alerts for archive job completion or failure, with non-production alerts prefixed by the mode name.
• Extended src/config.py to support notifications_enabled and notifications_slack_webhook_url settings, allowing notifications to be toggled and configured via environment variables.
• Updated src/api/README.md and main README.md to document notification setup and usage. [1] [2]

Worker integration

• Integrated notify_job_result calls into both submission_worker.py and retrieval_worker.py to send notifications on job success or failure, including relevant context such as IDs, drive name, and failure reasons. [1] [2] [3] [4] [5] [6]

Testing

• Updated integration tests in tests/test_chunked_workflow_integration.py to mock and assert notification payloads, ensuring correct notification behavior for both successful and failed jobs. [1] [2] [3] [4] [5]