Repository navigation
feat(models): evaluate xAI models directly via the Responses API - #50
Merged
Merged
Conversation
xAI was only reachable through `openrouter/x-ai/*`. `grok-*` and `xai/<slug>` ids now route to a new adapter calling https://api.x.ai/v1, reading XAI_API_KEY (or GROK_API_KEY). xAI serves an OpenAI-shaped /v1/responses, so this is the openai adapter with a different key, base URL and effort ladder. The Responses-API usage reader moves to models/__init__.py, shared by both. `grok-*` covers the published line; `xai/<slug>` is the escape hatch for models served under another name, passed through verbatim. Verified against the live API, which corrected two guesses made from the docs: - Temperature: docs say it is unsupported by reasoning models, but a reasoning model accepted `temperature: 0` and echoed it back. A benchmark wants determinism, so it is sent by default and withheld only from grok-3, the one family the docs name. A 400 mentioning temperature retries once without it rather than taking down a run; other BadRequestErrors propagate. - Output budget: passed through unchanged. An earlier revision floored it, to stop reasoning starving the answer. That bought nothing — the API does not truncate the answer to fit; a 2000-token cap returned 2047 tokens and a complete program — and cost a lot, because these models spend the whole budget on reasoning at ~36 tok/s. The floor put a call at ~15 min, past the configs' per-call timeout, after which the SDK retries twice and burns ~30 min per record silently. That is what a run that appeared to hang was doing. The effort ladder (none|low|medium|high|xhigh) is validated at parse time, so a ladder borrowed from another provider fails before the run, not partway through. 38 offline tests: routing, the suffix parser, and the request body against a stubbed client — endpoint, instructions vs system turn, budget pass-through, temperature policy and its retry, PNG data-url inlining.
BenchCAD
force-pushed
the
feat/xai-grok-adapter
branch
from
August 8, 2026 22:55
be28caf to
6005778
Compare
BenchCAD
changed the title
The timeout is the only effective bound on an xAI call. `max_output_tokens` looks like one but is not: it bounds the answer, not the reasoning, so a request capped at 2000 output tokens came back with 5360 tokens — 5326 of them reasoning — and reported status=completed. Raising or lowering it does not bound cost or duration. 900s comes from measurement. A high-effort call on a hard vision task normally takes 4-5 minutes, but re-running an identical request can blow past 7 minutes, so the latency tail is stochastic rather than a property of the prompt. The floor mirrors the anthropic and openrouter adapters, which already floor their reasoning paths; a larger explicitly configured timeout still wins. The openai SDK's default max_retries=2 is deliberately left in place. An earlier note in the openai adapter says its spirals re-spiral on retry, which would make retries pure waste — but that is not what xAI does here, where the same request completed once and stalled once, so a retry has a real chance of succeeding. Retries also cover the 429s any concurrent run will meet. The cost is that a hopeless call takes up to 3x the timeout, which is precisely why the floor is 900s and not an hour: bound the worst case in the timeout, not the retries.
Owner
Author
|
Added a 900s floor on the per-call timeout, and a note on why
Every call overshot its cap and still reported That leaves the timeout as the only real bound. 900s is chosen against the measured normal case of 4-5 minutes, with the openai SDK's default 109 tests pass. |
On every other provider a token cap is a ceiling. Here it behaves as a reasoning
target, so sending one makes a run slower rather than safer. Same record, high
effort, only this parameter varied:
max_output_tokens=16000 -> 32395 output tokens (32158 reasoning), 398 s
max_output_tokens absent -> 10089 output tokens (10022 reasoning), 134 s
Both returned status=completed with a complete CadQuery program, and note the
capped call overshot its own cap 2x. The parameter only ever bounds the visible
answer, which is a few dozen tokens and never approaches it — so it cannot serve
as a cost or duration control, and passing a large value to "remove the limit"
is the worst of the options: it triples reasoning and wall-clock for no gain.
The per-call timeout floor is left as the only real bound.
…3000s The 900s floor was calibrated on a 20-record sample where every call that hit it was rescued by an SDK retry, which made it look safe. Over a 299-record high-effort sweep it is not: successful calls have a median latency of ~1190s and a maximum of ~2520s, so 900s truncates more than half of the requests that are working normally. Each one then burns the SDK's full 3x retry budget (~2700s) and frequently still fails, so the "bound the worst case" floor was converting healthy requests into expensive failures. 3000s covers the observed maximum with margin. An explicitly configured larger timeout still wins, and max_retries is still left at the SDK default.
… memory
A run is dominated by waiting: each model call blocks for minutes while the
runner sits in a sequential for-loop, so a 17,900-record sweep is measured in
weeks. Concurrency is the obvious fix and the obvious implementation is wrong --
sizing one pool for both halves of a record either starves the API or kills the
machine, because the two halves cost completely different amounts of memory.
A model call is a socket plus a base64 image, a few MB, blocked on the network
for minutes. Scoring spawns a CadQuery/OCP subprocess (~0.5 GB resident) and
voxelises two meshes, and takes seconds. So `concurrency:` exposes two pools:
concurrency:
api_workers: 8 # concurrent model calls
score_workers: 2 # concurrent CadQuery runs -- bounds peak memory
A thread waiting on the API holds no scoring slot, which is what lets a large
API pool sit in front of a small scoring pool without deadlocking. The ratio is
forgiving: a couple of scoring slots keep up with dozens of API workers. Sized
naively at 64 concurrent scorers, the same work needs ~25 GB.
`api_workers: 1` is the default, so existing configs and their output are
unchanged; the sequential branch is kept verbatim rather than emulated.
benchcad_core/parallel.py also handles three things only visible under load,
each found by hitting it:
- VTK renders only on the main thread. Its Cocoa backend aborts the *process*
with an NSException if a worker builds a render window, and it leaks a
graphics context per render, so a few hundred in-process renders exhaust
memory. Ground-truth composites are therefore warmed up front, on the main
thread, into the on-disk cache the workers read.
- run_record persists results.jsonl read-modify-write, which silently drops rows
when threads interleave. The write re-reads and merges under a lock.
- A worker exception is reported against its record instead of ending the batch;
a record whose ground truth cannot be tessellated is dropped during the warm-up
and named, so a data defect is not mistaken for a model failure.
Also stops tracking Vision2Code/raw_responses.jsonl, which slipped into the
previous commit -- it is fetched model output, regenerable, and not source. It
carries no model identifier. lite_data/ is ignored and excluded from ruff for
the same reason test_data/ is: it holds CadQuery programs, not source.
BenchCAD
force-pushed
the
feat/xai-grok-adapter
branch
from
August 10, 2026 19:29
e33c805 to
5009ef6
Compare
Warming the ground-truth composite cache was not enough to keep VTK off the worker threads. Scoring renders a second image -- a preview of the *generated* solid -- and that call reached vtkCocoaRenderWindow from a worker and aborted the process with an NSException on the first concurrent run against the live API. The unit tests missed it because they stub the runner, so no real render site was exercised. The preview is explicitly non-critical to scoring, so rather than dropping it, `deferred_previews()` records the (step, png) pairs and `replay_previews()` draws them on the main thread once the pool has drained. Prompt building calls the same function without an output path and is served from the cache `prerender()` warmed, so it passes straight through and never reaches VTK. Verified end to end against xAI with api_workers=4 / score_workers=2: four records scored, four previews rendered, four rows persisted, 41.6s wall clock against 100.6s of summed API latency.
The adapter stopped sending max_output_tokens entirely, on the reasoning that a cap behaves as a reasoning target rather than a ceiling here -- a request capped at 2000 came back with 5360 tokens. That observation still holds, so the run config's budget is still not forwarded: a cap near the working range distorts the run rather than bounding it. Omitting the parameter outright was a step too far though. It leaves the request open-ended for no benefit, and it was briefly suspected of causing calls to hang. That suspicion was tested and is wrong -- at concurrency 1, with the endpoint degraded, 512000, 16000 and no cap at all time out identically, including the exact 16000 that was in flight during earlier successful runs. The cap is not a reliability knob. So send a ceiling far above anything observed (max ~33k reasoning tokens) as a backstop: it never binds, so it cannot shape the answer, but the request stays bounded instead of open-ended.
The adapter floored the per-call timeout, first to 900s and then to 3000s. Both values were derived from observed latencies that silently included the openai SDK's retries, so a call that really took ~350s looked like it took ~1200s and the floor kept being raised to "protect" work that was never at risk. Measured directly, without the SDK in the path: a successful call completes in 280-410s, and a call that has not returned by then does not return at all. The floor therefore never rescued a single request. What it did do was override the run config without saying so -- a fetch configured for 450s was silently issued at 3000s, so every dead call cost 9000s across its retries instead of 1350s, and throughput came out roughly 50x below what the same requests achieved when measured outside the adapter. The caller decides now. Task configs already carry an explicit `gen.timeout`.
Not for incremental output -- this endpoint buffers content and delivers it in a burst at the end either way. Streaming is for the keepalive. A high-effort call spends 300-1100s generating, and a non-streaming request sends no bytes at all for that entire window. Silent connections of that length do not survive the path to this endpoint: they are blackholed, which is why every failure was an indefinite hang rather than an error, a reset, or a 429. Paired A/B on identical requests, one at a time: 4/4 streaming versus 2/4 non-streaming, and the streaming set included a 1109s call that produced 40k reasoning tokens -- a pass no non-streaming timeout could have survived. That also explains the rest of the evidence: effort=none (5s) and low (30s) always succeeded, while medium and high did not; image size, record identity, token cap and concurrency were all irrelevant; and successful calls always ran at full speed rather than degrading, which is a connection-layer failure signature rather than a capacity one. The practical cost of getting this wrong was large: long-but-healthy passes were being cut off and recorded as endpoint failures, and the response was to keep tuning timeouts and worker counts against a problem that was not there. The keepalive fixes the timeout semantics too. httpx applies its read timeout per read, and a frame arrives every ~15s, so a generous per-call timeout no longer truncates a legitimately long generation.
BenchCAD
force-pushed
the
feat/xai-grok-adapter
branch
from
August 18, 2026 01:41
aa1b752 to
bdedcf6
Compare
The CHANGELOG entry and the module docstring were written against revisions of this branch that later commits reversed. They said the adapter floored the timeout, first at 3000 s and then at 900 s, in the same paragraph, and that a large output cap is the worst case, while the code sends a 512000-token backstop and has no floor at all. Describe what ships. `gen.max_tokens` is not forwarded, and a fixed backstop is sent in its place. Requests are streamed so their keepalive holds a long pass open. The caller's timeout is used as given. Temperature is pinned to 0.7.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
xAI was only reachable through
openrouter/x-ai/*.grok-*andxai/<slug>ids now route to a new adapter callinghttps://api.x.ai/v1, readingXAI_API_KEY(orGROK_API_KEY).xAI serves an OpenAI-shaped
/v1/responses, so this is the openai adapter with a different key, base URL and effort ladder; images and:reasoning=behave as they do elsewhere. The Responses-API usage reader moved tomodels/__init__.py, shared by both.Two model-id forms.
grok-*covers the published line.xai/<slug>is the escape hatch — not every model xAI serves is namedgrok-something, and a key may be ACL-scoped to a subset, so prefix-only routing would send a valid model to the wrong provider. The slug afterxai/is passed through verbatim, so no model list is baked in.How the adapter calls xAI (measured against the live API)
gen.max_tokensis not forwarded. A cap in the model's working range acts as a reasoning target rather than a ceiling. Withmax_output_tokens=16000a record used 32395 output tokens in 398 s; with no cap, 10089 tokens in 134 s. Both runs produced a complete program. A fixedmax_output_tokens=512000backstop, far above anything observed (~33k), is sent instead.gen.timeoutis used as configured. Two earlier revisions clamped it up, to 900 s and then 3000 s. Both values were sized from latencies that silently included the SDK's retries.none|low|medium|high|xhigh) is validated at parse time. A ladder borrowed from another provider therefore fails before the run starts, not partway through it.Also in this PR: concurrent evaluation
An optional
concurrency:block (api_workers,score_workers; defaults 1 and 2, i.e. the previous sequential behaviour) backed bybenchcad_core/parallel.py. The pools are separate because a model call is a socket that blocks for minutes, while scoring is a ~0.5 GB CadQuery subprocess that takes seconds. It also handles three failures that only appear under load:results.jsonlread-modify-write is serialised.Tests
The offline tests cover:
openrouter/x-ai/grok-4.5still goes to OpenRouter;instructions, the backstop, the temperature policy and its retry, and PNG data-url inlining;ruffis clean.Not covered
pricing.yamlhas grok-4.5 / grok-4.3 at the under-200k tier; xAI doubles both above 200k prompt tokens, socost_usdunderstates a long-context run. Unlisted models recordcost_usd: null.