Stream partial tool_calls argument deltas for coding agents (parity with vLLM/llama.cpp)

[tengotengo]

Problem

When serving a tool-calling model through the OpenAI-compatible endpoint with stream: true, tool calls are buffered until generation finishes and delivered as a single tool_calls delta containing the complete arguments. Coding agents that render live tool generation (e.g. pi-coding-agent) show nothing while the model is producing a tool call, then receive the whole thing at once after a delay.

Repro (exllamav3 backend, tool_format: qwen3_coder, Qwen3.8-Flash-Next EXL3):

[  1.92s] TOOL_CALL delta: idx=0 id=call_... name=get_weather args='{"city": "Amsterdam"}'
[  1.92s] TOOL_CALL delta: idx=1 id=call_... name=get_weather args='{"city": "Tokyo"}'
[  1.92s] [DONE]

Both deltas arrive at the exact moment generation completes, with full arguments. The reasoning and content channels stream normally throughout, so only tool calls lag.

Cause

This is the current design and it's documented. In endpoints/OAI/utils/chat_completion.py the streaming loop accumulates tool-channel text and only parses it on the final chunk:

else:
    full_tool += sub
...
generation["delta_tool_calls"] = ""
if finish_reason and full_tool:
    generation["delta_tool_calls"] = _parse_tool_calls(full_tool, tool_format, request_id)

and the _parse_tool_calls docstring states:

These are not choice indices; OAI enumerates the tool calls within each individual choice for the sake of streaming incomplete tool arg deltas, which we don't do here.

Request

Emit incremental tool_calls deltas - function name first, then argument JSON fragments - so clients can render tool generation live. This matches vLLM (--tool-call-parser qwen3_coder) and llama.cpp, both of which stream partial arguments.

I understand why it's non-trivial: the pseudo-XML format needs to be converted to a valid JSON arguments string incrementally, and whitespace stripping + coerce_param_value type coercion are only knowable at the closing parameter tag. A scheme that keeps the accumulated result byte-identical to the current end-parse:

• On the opening function=NAME tag: emit {index, id, function: {name}}
• On each parameter=k tag: emit {"k": " (or ,"k": " for subsequent params)
• String values: stream live with leading/trailing whitespace holdback (the same trick TagStreamParser already uses for partial tags)
• JSON-coercible values (arrays/objects/numbers): hold until the closing parameter tag and emit the coerced literal
• Keep the current authoritative _parse_tool_calls on the final chunk as the source of truth

Happy to contribute a reference implementation for qwen3_coder if there's interest - running exllamav3 + Qwen3.8-Flash-Next EXL3 in a pi coding-agent session, and this is the main UX gap vs vLLM for that workload. Saw #458 is also targeting pi integration.