Document what logprobs returns and bound it to the documented range

[Ranoobaba]

Have you read the Contributing Guidelines? Yes.

Issue #251

Describe your changes

What and why. Issue #251 asks how to get log probabilities for more than just the sampled token. The answer is the logprobs parameter, which takes the number of top tokens to score at each step, but nothing in the SDK says so. All four create() docstrings read:

logprobs (int, optional): Number of top-k logprobs to return

which reads as "how many logprobs come back" rather than "how many alternatives per token", and gives no range. The documented range is 0 to 20 for both chat completions and text completions: "An integer between 0 and 20 of the top k tokens to return log probabilities for at each generation step." Out of range values were only rejected by the server, so the round trip was spent before finding out.

The fix. Two parts.

1. Say what the parameter does, in all four create() docstrings (sync and async, chat and text), including the range.
2. Bound the field to the documented range:

logprobs: int | None = Field(default=None, ge=0, le=20)

on ChatCompletionRequest and CompletionRequest, so a bad value fails locally instead of after a request.

Why on the field and not in the validator. verify_parameters already exists on both models and was the obvious place, but a mode="after" model validator reports the whole request as the offending input. That puts the message list or the prompt inside the exception:

e.errors()[0]["input"]
# {'model': '...', 'messages': [{'role': 'user', 'content': 'PATIENT NAME: ...'}], 'logprobs': 50}

str(e) truncates it, so it looks harmless, but anything that serializes errors(), such as a crash reporter or structured logging, captures the prompt verbatim. Before this change the request went to the server and came back as an API error that did not repeat the prompt, so moving validation earlier should not hand user text to every logger. A field constraint keeps the error scoped:

e.errors()[0]["loc"]    # ('logprobs',)
e.errors()[0]["input"]  # 50

It also puts the range in the generated JSON schema, and it is less code than the validator branch. Two tests pin the prompt staying out of the exception, one per request model.

Known limitations

1. This is the first hard client side rejection of a pass through inference parameter in this SDK, and that is a judgement call. The parameter directly below it in the same docstring, logit_bias, documents its range of [-100, 100] and does not enforce it, and the same is true of temperature, top_p, min_p and n. The existing verify_parameters validator only ever warns. I enforced here because the range is documented for both endpoints and an out of range value can never succeed, so failing early is strictly faster. If you would rather keep the SDK consistent and permissive, say so and I will switch it to warnings.warn, matching repetition_penalty.
2. The range is hardcoded from today's API docs. If the service ever raises the cap, a pinned SDK will reject requests the server would accept, and the caller has no override.
3. The error is a pydantic ValidationError, not a TogetherException. Code shaped like except together.error.InvalidRequestError: around create() used to catch an out of range logprobs, because the rejection came back from the server. It will not catch it now.
4. The CLI reports it as a traceback, not a usage error. --logprobs is a plain click type=int on both chat.py and completions.py, so an out of range value reaches the model and surfaces as a pydantic traceback rather than a clean click message. Adding click.IntRange(0, 20) there would fix it, but it is a separate change in two more files and I kept this diff small.
5. Streaming plus logprobs is still not usable, and this PR does not fix it. ChatCompletionChoicesChunk.logprobs and CompletionChoicesChunk.logprobs are typed float | None, while the non streaming models use the LogprobsPart object. The same create() methods whose docstrings this PR rewrites also accept stream=True, so anyone following the new documentation with streaming on will hit a parse failure if the server sends the object shape on chunks. I had no API key to confirm the streaming wire shape, so I have not widened the type on a guess. Worth a follow up either way.
6. logprobs=True is silently accepted as 1. bool subclasses int, and pydantic accepts it in lax mode, so a caller coming from the OpenAI SDK, where chat logprobs is a boolean and the count lives in a separate top_logprobs field, gets top 1 rather than an error. Pre existing, unchanged here, noted because the new docstring says "integer".
7. The bound is checked at construction, not on assignment. The models do not set validate_assignment, so setting request.logprobs = 999 afterwards survives. Every create() builds the model in one shot, so this is not reachable through the public API today.
8. V1 is in maintenance mode (V2 lives in together-py). The docstring half is a plain documentation fix; the validation half is a behaviour change, so it is reasonable to want it in 2.0 instead. Happy to split them.

Testing

New file tests/unit/test_logprobs.py, 16 tests: 6 rejection cases across both request models, 6 acceptance cases including 0 and 20 as boundaries, one that logprobs stays out of the serialized payload when unset, two that the rejection does not carry the prompt, and one response side round trip.

Fail then pass: run against unfixed main, the 6 rejection tests fail with DID NOT RAISE ValidationError. On this branch the full offline suite is 222 passed, against 206 passed on main, and the difference is exactly the new file. No existing test changed.

Relation to open PR #452. #452 ("Fix chat completion request and logprob contracts") touches two of the same files and proposes declaring top_logprobs: List[Dict[str, float]] on LogprobsPart. The two are compatible and independent: #452 is response side typing, this PR is request side validation plus docstrings. I checked rather than assumed. They merge cleanly with git merge-tree, and all 16 tests here still pass with #452's typed field applied locally. If #452 lands first, nothing here needs changing. Whichever order suits you is fine by me.

The response side test deserves a caveat rather than a claim. top_logprobs is not a declared field on LogprobsPart, which declares only tokens and token_logprobs. It survives because the base model sets extra="allow", so the value is carried untyped and dumped back unchanged. That test pins the pass through and fails if the field is later declared with the wrong shape. It does not constrain what the server sends, and no source change in this PR affects that path. The test docstring says exactly that.

ruff reports one I001 on src/together/resources/completions.py, identical on main, in an import block this PR does not touch, so it is left alone per the contribution guide. black is clean and mypy --strict reports the same 14 pre existing errors in unrelated files before and after.

Everything runs offline. All 16 tests pass with socket creation and DNS raising and no API key set.

Addresses #251.

[broly-code-security-scanner]

Broly Security Scan

Note

✅ Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.

 

[Ranoobaba]

@blainekasten ready for review when you have time. This answers #251 by documenting what logprobs actually does in the four create() docstrings, and bounds the field to the documented 0 to 20. It merges cleanly with #452 and I confirmed the tests pass with that PR's field applied. One judgement call is flagged in the body: this is the first hard client side rejection of a pass through parameter, so if you would rather it warn than raise, say the word and I will switch it.