Skip to content

Extend Modal runner timeout - #496

Merged
msaroufim merged 1 commit into
mainfrom
investigate-modal-timeouts
Jun 18, 2026
Merged

msaroufim merged 1 commit into
mainfrom
investigate-modal-timeouts

Conversation

@msaroufim

@msaroufim msaroufim commented Jun 18, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Set Modal runner functions to a one-hour platform timeout.
  • Match kernelbot's in-function alarm timeout to the same one-hour envelope.

Root Cause

Production Modal logs for discord-bot-runner show B200 runner inputs being cancelled at Modal's default 300 second function timeout, including a cancellation at 2026-06-18 22:19:32 UTC that lines up with the 814188 failure window:

Task's current input ... hit its timeout of 300s
[modal-client] Received a cancellation signal while processing input ...
[modal-client] Successfully canceled input ...

The deployed runner functions were registered without a Modal timeout= argument, so the platform default could cancel the function before kernelbot's own timeout handling ran. Raising only the Modal platform timeout is not enough because modal_run_config still had an internal 600s alarm; this PR sets both to the same one-hour envelope.

Validation

  • uv run ruff check src/runners/modal_runner.py src/runners/modal_runner_archs.py
  • uv run python -m py_compile src/runners/modal_runner.py src/runners/modal_runner_archs.py
  • git diff --check

Deployment Note

After merge, the Modal runner app needs to be redeployed so the new function-level timeout=3600 is applied. Deploying only the Northflank service is not enough for this part.

@github-actions

github-actions Bot commented Jun 18, 2026 •

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  src/libkernelbot
  utils.py
Project Total  

This report was generated by python-coverage-comment-action

@msaroufim
msaroufim force-pushed the investigate-modal-timeouts branch from e997146 to 3624f8d Compare June 18, 2026 22:23
@msaroufim
msaroufim force-pushed the investigate-modal-timeouts branch from 3624f8d to a63c110 Compare June 18, 2026 22:25
@msaroufim msaroufim changed the title Extend Modal submission timeout envelopes Extend Modal runner timeout Jun 18, 2026
@msaroufim
msaroufim merged commit d66f6cd into main Jun 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant