Skip to content

LOU-V3: run a turn's tool calls in parallel (toolConcurrency) - #49

Merged
LinuxDevil merged 2 commits into
mainfrom
lou-v3-parallel-tool-calls
Oct 1, 2026
Merged

LinuxDevil merged 2 commits into
mainfrom
lou-v3-parallel-tool-calls

Conversation

@LinuxDevil

Copy link
Copy Markdown
Owner

What

The tool calls of one model turn now run concurrently. A new option caps how many run at once:

createAgent({ prompt, provider, tools, toolConcurrency: 4 });         // or 'unbounded' (default), or 1 = sequential
AgentExecutor.execute({ agent, input, provider, toolConcurrency: 1 });

Invalid values (0, -1, 1.5, NaN, 'all') throw: AgentExecutor.execute: 'toolConcurrency' must be a positive integer or 'unbounded', got 0. Use 1 to run a turn's tool calls one at a time, ... (same for createAgent). ToolConcurrency is exported as a type.

Why

Before this change the executor ran parallel tool calls from the model one after another. Models emit several calls in one turn so that they can run concurrently.

How

  • runToolCall() (toolCallExecution.ts) is split into a gate (onToolCall, LOU-U4 validation, preToolCall hooks, needsApproval) and the execution (execute, post hook, onToolResult). Behavior for one call is unchanged.
  • New src/execution/toolBatch.ts schedules a batch. It starts calls in call order, and each call starts only after the previous one has passed its gate and a slot under the limit is free. Results are recorded in call order as the completed prefix grows.
  • Building the tool-result message moved to agentRunState.pushToolResult().

Contract (TSDoc on ExecuteOptions.toolConcurrency + new "Parallel tool calls" section in docs/api-overview.md)

  • The transcript follows call order, whatever order the calls finish in.
  • tool-call events (and the gate) run in call order, just before each call starts. tool-result events arrive in completion order. With toolConcurrency: 1 the old interleaved order is kept exactly.
  • Approval: the first call that needs approval halts the batch. Calls before it run concurrently and are recorded. The run then pauses on that call exactly as before. Calls after it never start in this run.
  • If one tool fails, its siblings are unaffected. A propagating error (or a throwing hook) stops new calls from starting, waits for the running ones to finish, then rejects. No tool is left running detached.
  • Abort mid-batch: the run ends with finishReason: 'aborted'. Finished calls keep their results and the rest get "cancelled".
  • Checkpoints: written whenever the in-order prefix grows. With 1 that is per call, as before. With concurrency, a result is only persisted once every call before it is persisted, so a resumed run never re-runs a recorded call.

How verified

Ran after merging origin/main @ 89d199f:

  • npx tsc --noEmit: exit 0
  • npm run lint: ✖ 426 problems (0 errors, 426 warnings). All warnings were already there; none added in the touched files.
  • npm run build && npm run build --workspace=packages/create-loushy-agent: exit 0
  • npm run test:coverage: Test Files 99 passed (99) / Tests 1294 passed | 3 skipped (1297)
  • npm run fallow: exit 0, ✓ 0 above threshold · 2298 analyzed · maintainability 92.2 (good)
  • npm run docs:verify-snippets: all 34 snippet(s) type-check against src/; 5 also run cleanly

New tests in src/execution/parallelToolCalls.test.ts (mockModel + defineTool + deferred promises) cover:

  • real concurrency
  • transcript and provider request in call order when calls finish in reverse order
  • 1 is sequential with interleaved events
  • a limit of 2 with 4 calls never has more than 2 in flight
  • sibling failure isolation
  • propagating error: waits for siblings and starts no queued call
  • abort mid-batch
  • prefix-only checkpointing
  • approval at position 1/2/3 plus resume (earlier calls are not re-executed)
  • validation
  • createAgent pass-through

No existing test was changed.

Notes for reviewer

  • Known gap, not fixed here (LOU-U7): after resumeAfterApproval(), calls that came after the approval call in the same batch still never run and get no tool result. This was already the case before this PR. The approval tests assert it explicitly with a comment. Fixing it needs ExecutionSnapshot to carry the remaining calls and resume to run them, possibly with another approval pause.
  • needsApproval now runs in the gate, before execution starts. If it throws, the result is the same {error} outcome as before. The only difference is that its time is no longer counted in onToolResult latency.
  • Heads-up, unrelated to this PR: LOU-Z1, LOU-Z2: robust MCP schema conversion and result handling #46 (LOU-Z1/Z2) appears to have deleted LOU-Y2 skills (src/skills/*, docs/skills.md, the skills option) from main. This branch follows main, so it doesn't have skills either.

🤖 Generated with Claude Code

LinuxDevil and others added 2 commits October 1, 2026 09:34
When the model returned several tool calls in one turn, AgentExecutor ran
them one after another. They now run concurrently, capped by a new
`toolConcurrency?: number | 'unbounded'` option on AgentExecutor.execute()
and createAgent() (default 'unbounded'; 1 restores sequential execution;
anything else throws a descriptive error).

Contract (TSDoc on ExecuteOptions.toolConcurrency, docs/api-overview.md):
- Tool results are appended in the model's call order, not completion
  order, so the next provider request is deterministic.
- Calls start in call order; each call's tool-call event, onToolCall,
  validation, preToolCall hooks and needsApproval check run just before it
  starts, one call at a time. tool-result events fire on completion. With
  toolConcurrency 1 the event order is exactly the old interleaved order.
- Approval: the first call needing approval halts the batch. Earlier calls
  run (concurrently) and are recorded, then the run pauses on that call as
  before. Later calls never start (resume behavior for them is unchanged;
  LOU-U7).
- A failing tool gets its own error result; siblings are unaffected. A
  propagating error stops new calls from starting, awaits running ones,
  then rejects.
- An abort mid-batch keeps finished results and cancels the rest.
- Checkpoints are written whenever the in-order prefix of finished calls
  grows (per call with concurrency 1, as before), so a recorded call is
  never re-run on resume.

Implementation: runToolCall() is split into a gate (prepareToolCall) and
execution (settleToolCall); the new toolBatch.ts schedules a batch with a
small limiter. Tool-result message building moved to agentRunState.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…calls

# Conflicts:
#	README.md
#	src/createAgent.ts
@LinuxDevil
LinuxDevil merged commit 9089d8c into main Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant