Skip to content

fix(qm): keep job ends when the shutdown flush fails - #1036

Closed
janbjorge wants to merge 1 commit into
mainfrom
fix/keep-unwritten-job-ends
Closed

janbjorge wants to merge 1 commit into
mainfrom
fix/keep-unwritten-job-ends

Conversation

@janbjorge

Copy link
Copy Markdown
Owner

Fixes #1027.

Stacked on the supervisor refactor. That PR introduces the list. This one fills it.

Summary

  • A shutdown flush that fails keeps the job ends on QueueManager.unwritten_job_logs. Copy the list off before the next run, and write it with log_jobs on a live connection. A write after another worker owns the job is ignored.
  • pgq run --restart-on-failure writes those ends on the next connection before it dequeues. A new process does not have the list, so those jobs still run again, as ADR-0004 allows.

Testing

  • uv sync --frozen && uv run ruff check . && uv run ruff format . --check && uv run lint-imports && uv run mypy . && uv run pytest passed
  • test_run_keeps_job_end_when_shutdown_flush_fails keeps a successful end when the exit flush raises, and a later log_jobs removes the row
  • test_restart_writes_job_end_the_first_run_could_not writes that end on the restarted connection, and the handler does not run a second time
  • uv run ruff check, uv run mypy on the touched files passed

Checklist

  • I have read the Contributing Guide
  • I have added or updated tests
  • I have updated documentation if necessary

@janbjorge janbjorge added the Agentic Issue or PR written mainly by an AI agent. Required on all agent-authored contributions. label Oct 24, 2026
@janbjorge
janbjorge added this pull request to stack #1037 October 24, 2026 10:48
Base automatically changed from refactor/carry-unwritten-job-logs to main October 24, 2026 14:00
@janbjorge
janbjorge force-pushed the fix/keep-unwritten-job-ends branch from 99c2fbf to c4ca5b6 Compare October 24, 2026 14:00
A dead connection drops the status write, so a finished job stays
picked and runs again. The ends now stay on the manager, and an
in-process restart writes them on the next connection.

Fixes #1027.
@janbjorge

Copy link
Copy Markdown
Owner Author

Superseded: #1062 landed the same buffer hand-off, and the rest is ported in #1069.

@janbjorge janbjorge closed this Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Agentic Issue or PR written mainly by an AI agent. Required on all agent-authored contributions.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Job ends that cannot be written when QueueManager.run exits are dropped, so a finished job runs again

1 participant