fix: normalize null and missing fields in streaming chat deltas

[Ranoobaba]

Have you read the Contributing Guidelines? Yes.

Issue #160

Describe your changes

What and why. Issue #160 reports that the API returns an explicit null where the OpenAI streaming format leaves the field out, in three places:

1. choices[n].delta.tool_calls = null on text only chunks.
2. choices[n].delta.tool_calls[n].function.arguments = null on the first tool call chunk, where the name is given.
3. choices[n].delta.tool_calls[n].function.name = null on argument continuation chunks.

The root cause is on the API side. This PR does not change the wire, it stops the SDK from passing the problem on. The streaming delta model (DeltaContent) declares only content, so everything else rides on extra="allow". On current main that produces four separate problems, listed by how much they actually bite:

1. Tool call fragments come back as raw dicts, not the ToolCalls and FunctionCall models the non streaming ChatCompletionMessage uses. So the same tool call has two different shapes depending on whether you streamed it.
2. model_dump(exclude_none=True) does not strip the nulls nested inside those raw dicts, so {"function": {"arguments": None}} and {"function": {"name": None}} reach OpenAI compatible consumers even when the caller asked for nulls to be dropped.
3. delta.role raises AttributeError on continuation chunks. The API sends role on the first chunk of a response and leaves it out afterwards, so the same attribute works or raises depending on which chunk you are holding.
4. delta.tool_calls raises AttributeError when the field is absent. Worth being precise here: an explicit "tool_calls": null already parses to None on main, so this one does not fire on Together's own responses today. It fires on the OpenAI shape, where the key is left out. It is the same defect as 3, one field over, and it is the one the issue title points at, so it is fixed rather than left as a trap.

The fix. Two small classes in src/together/types/chat_completions.py, and ChatCompletionChoicesChunk.delta now uses the first:

class ChatCompletionDeltaToolCalls(ToolCalls):
    index: int | None = None

class ChatCompletionDeltaContent(DeltaContent):
    role: str | None = None
    tool_calls: List[ChatCompletionDeltaToolCalls] | None = None

That gives:

1. null and missing both parse to None for role and tool_calls, so the two are no longer distinguishable. FunctionCall.name and .arguments were already str | None, so the nested nulls normalize the same way once the items are validated.
2. Fragments validate into typed models, so streaming and non streaming agree, and it matches how openai-python types its deltas (ChoiceDeltaToolCallFunction.name and .arguments are Optional[str]).
3. model_dump(exclude_none=True) now recursively drops the API provided nulls.

Deliberate decisions:

1. index goes on a streaming only subclass. Streaming splits one tool call across chunks, so callers need index to reassemble them, but the non streaming ToolCalls has no such field. Subclassing keeps index out of ChatCompletionMessage.tool_calls dumps. A test asserts it in both directions.
2. role is typed str, not MessageRole. Chunks are parsed one at a time inside the streaming generator, so an unrecognised role value from a future model would otherwise raise part way through and end the stream.
3. No coercion of None into an empty string for arguments. openai-python also leaves absent fields as None; coercing would diverge from it and silently change is None checks.
4. Text completion deltas untouched. DeltaContent is shared with CompletionChunk, so the new fields are added on a chat specific subclass rather than the shared class.
5. No wire or request changes. Request serialization is unaffected, and the request bytes are identical before and after.

Sync and async clients share these models (ChatCompletionChunk(**line.data) at both call sites in resources/chat/completions.py), so both are covered.

Known limitations

1. The root cause is on the API side; this is client side hardening. Raw HTTP consumers and other SDKs still see the nulls on the wire. I did not have an API key to re-probe the live wire, so the payload shapes here come from the issue itself rather than a fresh capture. The SDK level behaviour reproduces deterministically from those payloads either way. Marked "Related to" rather than "Fixes" for that reason: the server side half is still open.
2. This makes the SDK stricter about the shape of streaming tool calls, and that can end a stream. Before this change, a fragment whose fields had the wrong type was passed through untouched as a dict. Now it raises a pydantic ValidationError, and because chunks are parsed one at a time inside the generator, that error ends the stream and the caller loses every remaining chunk, including plain text that had nothing to do with tool calls. Payloads that used to pass and now raise include function.arguments sent as a parsed object instead of a JSON string, tool_calls sent as a dict instead of a list, and id sent as an integer. This is the same strictness the non streaming ChatCompletionMessage.tool_calls has had all along, so it makes the two paths consistent rather than inventing a new rule, but it is a real behaviour change and worth a maintainer's judgement.
3. Source level breaks for anyone who worked around the bug. Fragments were dicts, so delta.tool_calls[0]["function"] worked; it now raises TypeError: 'ToolCalls' object is not subscriptable and needs attribute access. That is exactly the population that hit this issue. Also, a plain model_dump() without exclude_none=True of a text only delta now includes tool_calls: None and role, which is standard Pydantic optional field behaviour and matches ChatCompletionMessage and openai-python.
4. The new types are not exported from together.types.__all__. ChatCompletionDeltaContent is now the runtime type of every chat stream delta, so there is a case for exporting it, but that touches another file and I kept the diff to the fix. Happy to add it.
5. V1 is in maintenance mode (V2 lives in together-py), so this is scoped as a small maintenance bug fix. The underlying API cleanup this issue asks for is still worth doing server side.

Testing

New file tests/unit/test_chat_completion_stream_types.py, 8 tests using chunk fixtures shaped exactly like the issue's three instances, plus a missing field control, an undeclared field passthrough check, an exclude_none recursive dump assertion, and a non streaming regression control.

Fail then pass, with the new tests run against unfixed main:

6 failed, 2 passed
AttributeError: 'DeltaContent' object has no attribute 'tool_calls'
AttributeError: 'DeltaContent' object has no attribute 'role'
AttributeError: 'dict' object has no attribute 'function'
AssertionError: None survived in {'role': 'assistant', 'tool_calls': [{'index': 0, ... 'arguments': None}}]}

The 2 that pass on main are the deliberate controls: the non streaming path is untouched, and undeclared fields still flow through.

On this branch the full offline suite is 214 passed, against 206 passed on main, and the difference is exactly the new file. No existing test changed. ruff and black are clean on both changed files, and mypy --strict reports the same 5 pre existing errors in unrelated files before and after.

Everything runs offline. The tests build models from literal dicts, so no client is constructed, no key is read, and no host is named. I confirmed that by re-running them with socket.socket.connect, socket.create_connection, socket.getaddrinfo and ssl.SSLContext.wrap_socket all replaced with a raiser, after first checking the block itself fires.

[broly-code-security-scanner]

Broly Security Scan

Note

✅ Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.

 

[Ranoobaba]

@blainekasten ready for review when you get a chance. Small V1 fix for #160: the streaming delta now declares role and tool_calls, so an explicit null and a missing field stop behaving differently. 20 lines of source, 8 tests, all offline. The body flags one thing worth your judgement, that this makes the SDK stricter about malformed tool call fragments.