Run Python SDK handlers concurrently, not inline on the dispatch stream

[keithfz]

Problem

The Python SDK ran each handler inline on the dispatch stream loop:

for response in stub.Dispatch(...):
    result = handler_to_invoke(ctx)
    self.agent_stub.ReportInvocation(invoke_info)

The next message was not read until the current handler returned, so one slow handler stalled everything behind it. There is no threading anywhere else in the SDK, so the concurrency limit was exactly one.

Webhooks make this visible, and it is what was reported in CD-596:

1. The agent accepts a webhook and returns 2xx
2. It queues the invocation and sends it to the SDK
3. It starts a timeout timer
4. The SDK is still busy on the previous invocation
5. The timer fires — the agent logs invocation error: or context deadline exceeded

The Go SDK does not have this problem — it already runs a goroutine per invocation.

Change

Handlers now run on a bounded ThreadPoolExecutor, eight at a time by default and configurable with max_concurrent_handlers.

A semaphore stops the loop reading once every worker is busy. That is deliberate: the agent then sees its own queue fill, instead of this process queueing without limit.

Reporting moved onto the worker, so a failure there can no longer propagate into the dispatch loop and force a reconnect. It is caught and logged — otherwise the future would swallow it and the invocation would look like it vanished.

Note for handler authors

Two invocations of the same handler can now run at the same time, so handlers must be thread safe. max_concurrent_handlers=1 restores the old behaviour. Documented in the SDK README.

Tests

test_handlers_run_concurrently dispatches four invocations that block on a barrier. It passes only if all four are in flight at once, and fails against the serial version — verified by temporarily reverting the submit.

Two test-infrastructure fixes came with it:

• The mock gRPC server was not referenced anywhere after the fixture returned, so it could be garbage collected mid-test and stop listening.
• A MagicMock cannot serve a streaming RPC, so no existing test had ever exercised dispatch — the stream always failed and the client retried. The new test uses a real servicer.

Full suite: 9 passed, ruff clean.

🤖 Generated with Claude Code