Repository navigation
feat(agentic): Vision2Code with a Python sandbox and tools - #53
Merged
Merged
Conversation
Adds the tool-use setting from Anthropic's Vision2Code ablation: instead of emitting a CadQuery program blind from four rendered views, the model gets a working directory holding those views, renders its own solid through the same renderer that produced them, and compares the two before committing. Isolation is the point, so it is not optional. Model code runs in a container with no network, no host environment, a read-only rootfs, capped memory/CPU/ pids, unprivileged, and exactly one writable path -- the record's own work directory. If the image is missing the sandbox refuses rather than falling back to a bare subprocess: a degraded sandbox that still executes model code is worse than none, because it looks safe. Four things this run taught, each now enforced in code rather than remembered: - The bind mount must be probed. Docker does not fail when asked to mount a path its VM cannot reach; it mounts an empty directory instead, and every round then reports a missing file, which is indistinguishable from the model writing bad code. A whole 50-record run completed that way, every container execution returning rc=2, and the result read as a large agentic gain when the sandbox had never run anything. Sandbox construction now probes and refuses. - The sandbox must execute under the same OCP compat shim as the scorer, imported rather than copied. Without it every .faces/.edges selector raises, so the model's program failed in the sandbox while the identical program succeeded at scoring -- the exact inverse of the guarantee the sandbox exists to make. - Images shown to the model must outlive the per-round reset. Reset ran before the observation was assembled, so every render the model produced was deleted before it could be seen; the round then died reading a missing file. Runs are archived to the log directory and the archived copies are what get shown. - The rounds are for investigation; answering is separate. A model that spent its whole budget measuring was cut off having submitted nothing, because the final round's output was never shown and no turn ever asked for a program. Every record in the first working run ended that way. Logs live outside the mount, so a reset cannot erase them and a failure can be diagnosed without re-running the model. STEPs are archived alongside, which is what makes the "fall back to the last solid the model built" path reachable at all.
Three things the first complete run exposed. The submission guard checked that the model had executed *something*, which is a far weaker condition than having looked at its own geometry. An episode ran nine rounds of pixel forensics on the target image, produced no render in any of them, and submitted -- passing the check the whole way. The guard now requires a round to have produced an image, which is the mechanism this setting exists to test. The model also spent its first round rediscovering the directory: listing files, printing image sizes, reading tools.py. All of that is knowable in advance, so the prompt now states the exact sizes, the quadrant ordering, and the full tool signatures, and says not to spend a round on discovery. It also describes how to spend the rounds -- build something whole immediately, render, diff, then fix the largest disagreement -- because the rounds are worth little if the model does not know what they are for. Finally, `timeout` did not bound anything. The SDK applies it per read, and the streaming keepalive resets that clock every ~15s, so a request that had stopped progressing could trickle frames indefinitely: one call ran 3685s under a 1800s timeout and returned a normal-sized response, holding a worker for over an hour. Elapsed time is now checked against `timeout` as a wall clock, which is what the parameter has always appeared to mean. Measured against single-shot on a paired subset; the gain is concentrated on records single-shot scored poorly and reverses on records it already got right.
…nvironment The agentic runner needs the model to see its own previous programs as its own turns, not as text quoted back at it, so `call_model` grows an optional `turns` list. When given it replaces `user_text`/`image_paths` entirely. Adapters that have not implemented it raise NotImplementedError naming the ones that have, rather than silently dropping the history and returning a plausible single-shot answer. Separately, `execute_cq_to_step` ran model-written code with a copy of os.environ, which handed every provider API key in the parent process to code the model authored. That was already the wrong default for a single-shot program; it is indefensible once the model is told it is working in a loop and that its code will be executed. The subprocess now gets only what the interpreter needs to start.
`tools.render` used the renderer's default of 128 pixels per view, which is the lite set's geometry and produces a 268x268 composite. The public set's targets are rendered at twice that, 524x524. So on the public set the model's own render came back at half the target's size, and the comparison it is there to make -- subtract the two arrays -- raised a shape error every time. It is the failure mode with the worst signature: nothing errors at the harness level, the episode completes, a score comes out, and the one mechanism the setting exists to test never runs. In a 6-hour run over the public set, 121 of 127 records hit it. The per-view size is now derived from the target actually seeded into the directory, so a record is matched to its own dataset rather than to whichever one the renderer was vendored from. Verified on both: public 524x524 vs 524x524, lite 268x268 vs 268x268, numeric diff working in each.
Run output landed next to the source as an untracked file, one stray `git add -A` away from being committed. Results belong to a run, not to the repo.
BenchCAD
force-pushed
the
feat/xai-grok-adapter
branch
from
August 18, 2026 01:41
aa1b752 to
bdedcf6
Compare
BenchCAD
force-pushed
the
feat/agentic-sandbox
branch
from
August 18, 2026 01:41
2847922 to
4780167
Compare
`_step_to_normalized_mesh` puts every shape in a unit cube centred on LOOKAT, so no vertex is farther from the centre than the half-diagonal sqrt(3)/2 = 0.866, and the cameras look down the cube diagonals where that bound is approached. The parallel scale was 0.55, which clips anything blockier than a rod: over the 392 preference-lab references, 117 of them (30%) projected past the frame, the worst needing 0.746. Vision2Code builds the image the model is asked to reconstruct with this same renderer, so those parts were posed as questions that could not be seen in full. Nothing noticed, because a clipped render is still a valid PNG. 0.90 clears the bound with a small margin; the bound is geometry, not a fit, so it holds for shapes not yet in the corpus, and every extra unit shrinks the part. Also releases the render window. `arr` was a view onto VTK-owned memory that Finalize frees, and without Finalize the process accumulates one graphics context per view and dies 121 shapes in, every time, in an uninterruptible wait with no error and no traceback. Tests project the vertices the way the camera does and compare extent to viewport — no VTK, no window, so they run anywhere.
The suffix went from 2599 characters to 993. Gone: the round-by-round strategy advice, the claim that diffing a render beats reading the target's pixels, the scolding about spending a round on discovery, and the threat that a submission which never rendered is rejected. Left: what each fenced block does, the file list, the tool signatures, one sentence that a render is directly comparable to the target, and the round budget. The code gate that enforced that threat is gone too. Requiring a render before a submission is a strategy requirement dressed as a protocol one — it forces a round to be spent producing an image whether or not that is the best use of it. Whether the tools are worth using is the measurement, not the premise. Measured against the long prompt on a paired subset, the difference in score is within noise; the short prompt costs about half the tokens. So this is not a score improvement, it is the same result for less, and one less thing the harness asserts about how the model should think. Also: the round budget stated in the prompt now follows the budget actually enforced. It came from the module constant, so a per-call max_rounds of 25 told the model it had 10 while the observations counted to 25. And tools.crop's docstring quoted a 512x512 composite, which is neither dataset's geometry.
Resetting after every round forces each block to rebuild everything it needs from scratch. The model cannot keep a helper script, a measurement it computed, or the solid it built last round — so an episode stops being a working session and becomes ten independent attempts that happen to share a transcript. That is not the setting the ablation is about, and it is the opposite of how the agent harnesses this is modelled on treat a workspace. The directory is now seeded once and persists for the record's whole episode. The prompt says so, since it previously promised the reverse. Logs still live outside the mount, so nothing the model overwrites is lost for diagnosis, and the "fall back to the last solid built" path reads them rather than the working directory. Every agentic result collected so far used per-round reset and is not comparable to anything measured after this; the arms need re-running together.
The sandbox vendored `lite_renderer.py`, which draws a 60-degree perspective. The corpus targets were drawn by `views.py`, a parallel projection. So every comparison the model made was between two different projections of the same solid, and no amount of fitting could close that gap — measured against a stored target, the sandbox's own render of the ground-truth STEP came back at RGB MAE 56.2. It now vendors `views.py` and pins the parallel scale to the value the corpus was rendered at. Same STEP, same measurement: MAE 2.58, silhouette IoU 0.985, 3.6% of pixels differing and only at edges. The residue is the rendering backend — the targets were drawn on a host GPU stack, the sandbox draws through OSMesa in the container — and the same module renders MAE 0.00 against the targets when run on the host. The scale is pinned in the generated tools.py rather than inherited from the module, because the module's value has already moved once to fix a clipping bug. If it moves again, the model's render silently stops matching the image it is asked to reproduce, and that comparison is the only reason the sandbox exists. The prompt said "the same four cameras, the same layout and the same colour", which was true and beside the point; it now says the same renderer and the same projection.
`cost_usd` charged every prompt token at the input rate. Providers serve repeated prefixes from a cache at a fraction of it — xAI reads cache at $0.30/M against a $2.00/M input rate — and a multi-turn loop resends its whole transcript every turn, so most of its prompt is a cache hit. Ignoring that does not add noise, it systematically overstates the cost of exactly the setting this benchmark is measuring, and the comparison it feeds is "is the agentic loop worth its cost". Verified against a live call: 212 prompt tokens of which 128 cached, 21 output. Billed $0.000330. Full-rate arithmetic says $0.000550; cache-aware arithmetic says $0.000332. `usage` now carries `cached_tokens` for both the Responses and Chat Completions shapes, and `cost_usd` takes it as an optional argument. Omitting it bills everything at the input rate, which stays a correct upper bound — results collected before this can still be costed, they just read high.
…cript Every agentic number so far came from a script outside the repo, which means nobody else could produce one. `run_record` now takes `agentic` and `max_rounds`, `main.py` reads them from the config, and `configs/agentic.yaml` is a working example. Same records, same executor, same voxel IoU as the single-shot path — the only difference is whether the model gets a working directory or one shot. Off by default. A harness that silently degrades to single-shot when the container image is missing would report one setting's score under the other's name, and that failure has already happened here in a subtler form. The result row carries the round count and the sequence of actions, so a run can be read back for whether the model actually used the sandbox rather than submitting from the first turn.
Three lint errors, all mine, all from edits made without running the linter the CI runs: - `SYSTEM_SUFFIX` kept its `f` prefix after the round count moved from an f-string interpolation to a `.format()` placeholder, so it was an f-string with nothing to interpolate and every brace was doubled for no reason. - the mode line used percent formatting. - an import block went unsorted. `uv run ruff check .` is what CI runs; running it locally is the fix for the habit, not just for these three.
`subprocess.run(timeout=)` kills the local docker CLI. The container it started keeps running, and `--rm` only removes a container that exits on its own, so model code that hangs leaves one behind forever. The containers were unnamed, so there was nothing to kill even if we had tried. A 100-round batch made this visible: 52 of 58 live containers were older than the 300s command timeout, the oldest 40 minutes, and the VM went from 10.7 GB free to 3.8 GB on its way to an OOM that would have looked like the model writing bad code. Every earlier run leaked too, just slowly enough to finish first. Each command now runs in a container named for its record and round, and a timeout kills it by name.
cadquery 2.3 calls TopoDS_*.HashCode; cadquery-ocp 7.9 removed it, so every .faces()/.edges() selector raises without a patch — and that is most of CadQuery. The patch existed only as a string the harness prepended to each program it executed. Our own runs worked; the image did not. A plain `python3 -c` in this container could not use a selector at all, which is exactly what a second harness hit the moment one was pointed at it: it spent 82 steps and three dollars fighting an environment we had quietly been fixing for ourselves. It is a property of the environment, so it lives in the image now, applied through sitecustomize at interpreter start. OCP imports in ~0.3s and only when something asks for it. Verified in the rebuilt image without importing anything of ours first: faces() and edges() both work.
Pushed the previous commit without acting on ruff's output, in the same commit that said running ruff locally was the fix for the habit.
…orks to Every agentic number in this branch was collected at 10 rounds, where the model hits the cap 98% of the time. That is not a measurement of the setting, it is a measurement of the cap. Given 100 rounds it stops on its own: median 70 used, 4 records in 50 reaching the cap. Paired over the same 50 records, 10 -> 100 is +0.266 mean IoU (95% CI [+0.176, +0.360], 43 wins to 7) — more than twice the effect of giving the model the sandbox at all, which was +0.108. The cost is real and now documented in the config rather than implied: 55k tokens per record at 10, 3.34M at 100, $0.15 against $7.25. 25 rounds keeps about half the gain for a seventh of the cost and is the sensible dial-down. The old config comment claimed 25 rounds was measured no better. That was true of the sandbox that reset every round, where extra rounds only bought extra restarts; it stopped being true when the directory started persisting, and the comment outlived the fact.
The sandbox caps stdout at its last 4000 characters and keeps the tail on purpose. _observation then took the first 2000 of that, which inverts the choice: the model's way of working is to sweep parameters in a loop and print the winner on the final line, so the head of a long output is the part that carries no answer. Measured over 3470 stdout samples from completed episodes: 30% were long enough to be cut, and 36% of those lost the BEST line the round existed to produce. mini-swe-agent, which keeps head and tail with an elided count, leads us on exactly the low-baseline records whose sweeps print the most. The sandbox's 4000-character tail is already a bound, so removing the second truncation leaves the observation bounded and keeps the line that matters.
A file the model writes into the work directory was treated as an image on the strength of its name and mtime alone. Two kinds get through that way and both come back as 400 invalid_image, which is not retryable -- the retry resends the same turns with the same bad image, so all five attempts fail identically and the episode ends mid-investigation. A container killed mid-write leaves a zero-byte .png: eight across 47k renders. And a crop whose margin runs off the left edge leaves a structurally perfect one: numpy reads a negative slice start as an offset from the end, so `edge[rmin-5:rmax+5, cmin-5:cmax+5]` with cmin=4 selects a single column and writes a valid 1x173 PNG. Between them they truncated six records of a 299-record run, one at round 19 of 100. Require a real header, an IEND to show the writer finished, and both dimensions at least 8px -- well under the smallest legitimate render seen (74x48) and well over the degenerate ones.
… was cut The flat 4000-character tail cap was cutting 23.4% of rounds, measured over 6049 archived stdouts. The distribution is bimodal for a reason: median 556 characters when the round takes a couple of measurements, straight into the cap when it sweeps parameters. So the cap fires precisely on the rounds whose output is a search and skips the ones that print two numbers. Take mini-swe-agent's policy verbatim -- pass through under 10k, otherwise head 5000, tail 5000, and a count of what went missing. Head and tail keep both the setup and the winning line, and the count tells the model it was truncated, which a silent cut never did. Matching the number also removes a confound from the head-to-head: mini was reading 2.5x the observation on the same records. Costs roughly 25k additional context tokens over a 100-round episode against the 3.5M an episode already spends.
…d of doing it Persisting the raw replies settled what four earlier guesses could not. Across 109 archived calls, 13.8% carried no parseable block, and every one of them is short prose -- "I'll inspect the images and craft CadQuery to match the bolt geometry." Not one tag mismatch, not one unfenced program. The model was answering conversationally, and 62% of first rounds went that way. The prompt caused it. The base system prompt says to output only a fenced block and no prose; the agentic suffix then said "you are working in a directory, not answering in one shot" and documented the two tags in a table whose ``` markers open two fences and close neither. Read together that licenses a conversational reply, and the model took the license. The suffix now states the per-reply contract as an instruction -- one fenced block, beginning at the opening fence, no preamble -- and describes the tags in prose so the demonstration of fencing is not itself malformed. Three supports around it: - The nudge restates the contract instead of saying a block was not found. The failure self-reinforces, because a rejected reply stays in the context as an example of itself, so the correction has to carry the format. - A reply with no block no longer costs a round. It produced no work; charging the budget for the harness's own parsing problem is how 12 of 299 records burned all ten rounds and never ran a line. Recovery gets its own budget of five extra calls. - The fence regex accepts any tag, `cadquery` meaning submit and everything else meaning run. Insurance rather than a fix, given zero tag mismatches in the sample, but a tag is a free thing to get wrong and submitting is the act that ends an episode, so ambiguity resolves toward running the code.
…nced text Persisting the raw replies showed the model opening a run with prose -- "I'll inspect the images and craft CadQuery to match the bolt geometry." -- and nothing else: 52% of first rounds, 11.9% of all calls over 185 samples, with zero tag mismatches and zero unfenced programs. Every text-side fix attacks the symptom. A fenced convention lets a reply that only announces work count as a complete turn, and no wording makes that structurally impossible. Under a schema it is impossible, and the run this switch produced says exactly why. The model still writes the preamble -- the first tool call of the smoke test carries the text "I'll inspect the target views and measure key proportions so the CadQuery solid matches the render." Identical behaviour. But now it rides alongside a run_python call instead of replacing it. Over 32 calls in a 12-record trial: zero replies without a tool call, 4 of 4 first rounds calling a tool, against 52% failing before. Two tools, run_python and submit, both taking `code`. The interpretation the tag used to carry is now the function name, so `_blocks` and its regex are gone, and so is the tolerant-parsing insurance added an hour ago against a failure mode that turned out not to exist. Mechanics worth noting: - Turn grows `tool_calls` and `call_id`; call_model takes `tools` and widens its return to ToolCompletion. Separate type rather than a third field, because every existing caller unpacks two names. - The Responses API treats a call and its result as top-level input items matched by call_id, not as message content. Nesting them is accepted by the request validator and then ignored by the model, which reads as amnesia. - Every call in a response needs a result, including ones we decline to run, or the next request is rejected for an unanswered call. - Images cannot ride on a tool result, so they follow as a user turn. Without that the model is told an image exists and never sees it, which undoes the point of the sandbox. - A reply with no tool call still does not cost a round, and the nudge now names the tools. Measured at zero so far, but the budget should not pay for a protocol failure if one appears. This also removes the last structural difference from mini-swe-agent, whose comparison had it enforcing format through a schema while this side asked in prose.
10 of the first 353 executions of a run hit the 300s cap. They were on ten different records, one each, and every one was building geometry rather than spinning -- so the cap was censoring work, not catching runaways. A round lost that way is not fatal, since the model is handed "timeout after 300s" and carries on, but it is a round that produced nothing. The same value bounds the final scoring execution, where the cost is worse: a submitted program slower than the cap scored zero for being slow rather than for being wrong. What 600 recovers is not measurable from here -- a 300s cap censors the distribution it would need to be sized against, and a program killed at 300 may well have wanted 900. It is a starting point sized to the observed rate, and the timeout rate is worth watching after the change rather than assuming it went to zero.
An episode that spends its whole budget without submitting has failed to manage that budget, and that is part of the task. The harness was making one extra call to ask for the program anyway, which is a rescue no comparable agent gets: mini-swe-agent hits its step limit, leaves no answer file, and scores zero. The rescue is also dead code at the budget actually in use. It fired 24 times in 299 records at 10 rounds -- a budget 98% of episodes exhausted -- and the answers it extracted were worth 0.027 IoU over the single-shot baseline, so it was rescuing near-nothing even then. Across 104 records at 100 rounds it fired zero times: 9 reached the cap and all 9 submitted on their own, because the last observation tells the model the round was its last and the model acts on it. Telling the agent its budget is the honest version of this, and that part stays. What it did buy was the whole measured advantage over mini-swe-agent. Paired over 48 records the two harnesses differed by -0.0101 overall, and by -0.0699 once mini's five unsubmitted records were set aside -- so every point of the apparent lead came from a mechanism the comparison arm did not have.
A submission that failed to execute fell back to the newest STEP anywhere in the episode log. That file is whatever geometry the model happened to build last, which need not be what it put forward -- a half-finished probe from round 3 could be scored as the answer while the program the model actually chose was discarded for raising. It was also the second of two rescues this arm had and the comparison arm did not. mini-swe-agent reads the single file its protocol names and scores zero when that file is missing or broken; five of fifty records ended that way. An episode here could submit nothing, or submit something broken, and still score off an artefact it had abandoned. Removing it costs whatever it was quietly earning, which is not measurable from the runs on disk -- no field recorded which STEP was scored. The replacement for that measurement is that there is now nothing to measure: what gets scored is what submit was called with.
…es apart Two changes from the same record. cable_routing_panel_000076 ran 21 clean rounds and died on the 22nd against a 27x16 crop the model had written, then scored zero because nothing had been submitted yet. The guard now bounds the pixel count rather than the sides. Its two earlier revisions each took their threshold from whatever had just broken -- a header check after the zero-byte files, 8px per side after a 1x173 edge render -- and 27x16 clears both while still being refused. xAI states the actual rule in the rejection: "Image has 432 total pixels (27x16), which is below the minimum of 512 pixels." So this is that number instead of a fourth guess at it. Three more sub-512px files were already sitting in the work directories of in-flight records, waiting for the round that would reference them. The zero is the second problem. An episode with no program has two unrelated causes -- the model never submitted, or the provider stopped answering -- and scoring both as zero measures our API reliability alongside the agent. At 100 rounds the provider cut short 11 of 104 records, and under the fallback that just got removed those scored 0.5146 on average, so the policy is worth about 0.055 of mean IoU: the same size as the entire gap against mini-swe-agent. run_agentic now returns the provider's error as `api_err` and the runner prefers it, which is what a caller needs to re-run the record rather than record a result for it. mini-swe-agent has no matching exposure -- its failures are LimitsExceeded, which is the agent's own budget and does belong in the score.
The observation for round max_rounds ended with "That was the last round. Call submit now." and then the loop exited on that same observation, so the model was told to do something and never asked again. Dead text that costs tokens and reads as an instruction. The fix is to remove the notice, not to grant a turn for it. mini-swe-agent raises LimitsExceeded before the call that would exceed its budget -- no warning, no closing call, submission empty, and five of fifty records ended exactly that way at n_calls=100 with cost still under half the cost limit. Our loop already has those semantics: submit is available on every round, and spending them all on investigation is a choice. Announcing the last round, or adding a turn to answer in after the work is over, would be a mechanism the comparison arm does not have -- which is what the forced final call was, under a different name.
…uter path
Three things the ablation runs needed, all of which had been living only in
working copies on the run host.
`prompt.py` gains two alternative statements of the camera geometry, chosen by
`BENCH_VIEW_HINT` so an arm differs in that block and nothing else:
(unset) all four camera positions, as before -- byte-identical output, so
the existing 299-record baseline is untouched
anchor only the top-right [1,1,1] camera; the rest are said to be corner
directions without saying which is where
perturb same, but worded for targets whose three undisclosed views were
re-rendered 15 degrees off-axis; the magnitude is withheld because
naming it hands over the recipe for searching the offset back
The block is load-bearing rather than decorative: 86% of the 299 baseline runs
transcribed those vectors into their own code, and 47% of the `anchor` runs
reconstructed the withheld ones anyway -- which is the result the arms exist to
measure.
`openrouter_adapter.py` grows multi-turn and native tool calling, so the agentic
runner is no longer xAI-only. The harness states tools in the Responses API's
flat form and chat completions wants them nested under `function`, so they are
converted here rather than at each call site. Two failure modes are handled
because they cost whole runs elsewhere: a model that rejects `temperature` gets
one retry without it, and a reply truncated at the token ceiling raises instead
of looking like "the model chose not to call a tool".
OpenCode Zen rides the same adapter behind an `opencode/` prefix -- same wire
format, different base url and key -- rather than earning a near-duplicate file.
`BENCH_NO_IMAGE_FEEDBACK` drops the user turn that carries the PNGs the agent's own code wrote. Everything else -- prompt, tools, budget, the tool result text itself -- is byte-identical, so an arm run with it differs in exactly one message. Worth measuring because the render reaches the model twice: once as pixels its own code reads and reduces to a number, and once as a picture attached here. Only the second is ablated. Measured over 100 paired records the difference is -0.018 IoU, 95% CI [-0.062, +0.027] -- and mini-swe-agent, which never receives a render at all, ties with us on the same task.
…auses 030e071 widened the viewport to 0.90 as a clipping fix. It is also a change of benchmark input. views.py draws the image every single-shot Vision2Code record is posed with, and every published number, ours and the providers', was measured at 0.55. At 0.90 each part renders at 61% of its former size, so scores from this repository would stop matching the scores they are compared against. Restore 0.55 and keep the render-window release from the same commit, which leaves the pixels untouched. The clipping is real: table_000328 needs 0.746 and clevis_000428 0.710. The checks that prove it stay, as strict expected failures, so they fail the day the scale changes and the marker cannot outlive the fix. The wider framing belongs in a versioned input change with re-rendered dataset views (#54). The sandbox was already pinned to 0.55 (GT_PARALLEL_SCALE), so the agentic render tool and its targets agree either way.
The harness only ever looked for `benchcad-sandbox:arm64`, and the amd64 Dockerfile said nothing about tags. On an x86-64 host, which is most of them, following the instructions gave "sandbox image not found". Pick `benchcad-sandbox:arm64` on arm64 and `benchcad-sandbox:amd64` elsewhere, say which Dockerfile builds which, and let BENCH_SANDBOX_IMAGE override the choice. The existing arm64 setup is unaffected.
Five per-run outputs of the mini-swe-agent comparison were committed at the Vision2Code root. They are run data, like the agentic_*.jsonl files already ignored. Remove them and ignore the pattern.
The module docstring still described the text-fence protocol that 1e689e1 replaced. It said turns were orchestrated rather than tool calls, and that the model submitted with a ```cadquery block. It now describes the two native tools, and that an unsubmitted episode scores zero. Add a Vision2Code README section with what the model gets, how to build the image and run the config, and the ablation switches. Add CHANGELOG entries for the setting, the scrubbed execution environment, the render-window fix and the framing kept on purpose.
Measured over a 399-part code_gen sample by projecting each part's vertices the way the camera does: 136 (34%) extend past the 0.55 frame in at least one view, the worst needing 0.784, and none reach the 0.866 bound.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Vision2Code currently asks the model to read four rendered views and emit a
CadQuery program blind. This adds the tool-use setting: the model gets a working
directory holding those views, renders its own solid through the same renderer
that produced the target, crops either image to check a detail, and compares
them numerically before committing.
The mechanism turned out to be measurement rather than inspection, and reading
the transcripts is what showed it. The model does not eyeball its render against
the target — it reads
_render.py, writes a parameter sweep, scores eachcandidate in code, prints
BEST sx=1.02 sy=0.98 IoU=0.999227, and converges onthat number. 72% of programs compute a metric; 57% write a sweep loop.
The model drives this through two native tools,
run_pythonandsubmit.This revision (before merge to
main)main, after feat(models): evaluate xAI models directly via the Responses API #50 merged. This brings in the two commits thatadd
BENCH_NO_IMAGE_FEEDBACKandBENCH_VIEW_HINT, together with the OpenRoutermulti-turn / tool-calling path.
views.PARALLEL_SCALEto 0.90 as a clipping fix, butviews.pydraws the imageevery single-shot record is posed with, and every published Vision2Code number
was measured at 0.55. At 0.90 each part renders at 61% of its former size, so
scores from this repository would stop matching the scores they are compared
with.
code_gensample (34%) extend pastthe frame in at least one view, the worst needing 0.784.
to a versioned input change with re-rendered dataset views (Vision2Code: HF dataset ships 128 px views, the GitHub runner renders 256 px #54).
GT_PARALLEL_SCALE), so the rendertool matches the targets exactly (RGB MAE 0.00).
benchcad-sandbox:arm64onarm64 and
benchcad-sandbox:amd64elsewhere, withBENCH_SANDBOX_IMAGEtooverride. Before, an x86-64 host following the instructions got "image not
found".
Vision2Code/root andthe pattern ignored.
now describes the native tools. The Vision2Code README gains an
agentic-setting section (build, run, ablation switches), and the CHANGELOG
has entries.
Results
Paired over the fixed 299-record public subset,
xai/grok-4.5:reasoning=high,100 rounds:
By single-shot difficulty:
Median 39 rounds, 2,365,815 tokens per record, zero episodes hit the 100-round
cap — the model decides it is finished and submits rather than spending the
budget.
Three earlier findings this run overturns
An earlier revision of this description stated three results from pilot runs.
The full run contradicts all three, and the reason in every case is that the
pilots were measuring harness defects rather than the setting.
"The gain reverses where the baseline is strong." It did: the previous
harness scored −0.284 on records above 0.8 and won only 21% of those pairs.
That was the loop failing to be a ratchet — nothing let the model decline to
change an answer that was already right. It is now +0.2214, 78 wins to 5,
and every difficulty band is positive.
"Gate the setting on a weak first attempt." That followed from the reversal.
With no reversal there is nothing to gate against, and gating would now discard
the +0.2214 the easy band earns.
"More rounds is not the missing ingredient." Measured again on the fixed
harness, paired over 50 records against the same 10-round arm: 25 rounds
+0.1231 (CI [+0.066, +0.190]) and 100 rounds +0.2661 (CI [+0.176, +0.362]).
Both clearly positive. The original null came from a 10-round arm losing a
median of 2 rounds to a parsing bug before any work started.
What the commits fix
Nine defects, found by reading transcripts rather than by guessing. Each is
documented at its site; the ones that moved the number:
4000 characters on purpose; the caller then took the first 2000 of that. The
model prints its answer on the final line, so the half being discarded was the
half that mattered — 30% of outputs were cut and 36% of those lost the
BESTline. The budget is now 10k with head, tail and an elided count, matching
mini-swe-agent so the comparison is not confounded by observation size.
only prose — "I'll inspect the images and craft CadQuery to match the bolt
geometry." — and 12 of 299 records spent all ten rounds that way without
running a line. Persisting the raw replies is what settled it: 185 samples, no
tag mismatches, no unfenced programs, just prose. Under a schema the model
still writes that sentence, but it now rides alongside a
run_pythoncallinstead of replacing it. 11,000+ calls since, zero without a tool call.
killed mid-write, and a 1×173 image from
edge[rmin-5:rmax+5, cmin-5:cmax+5]where numpy read the negative start as an offset from the end. Both return
400
invalid_image, which is not retryable — the retry resends the sameimage. The guard is sized to the rule the API states rather than to whatever
had just broken:
w * h >= 512.when it never submitted, and scored the newest STEP anywhere in the log when
the submitted program failed to execute. Neither has a counterpart in
mini-swe-agent, and together they were the whole apparent advantage over it.
Against mini-swe-agent
Same 50 records, same model, same image,
step_limit=100,cost_limit=$10:The previous harness lost significantly; this one is indistinguishable. The
two agents have no measurable capability difference on this task — every
explicable difference was our own engineering.
One caveat on the mini numbers, because it changes their reading. Five of its
records hit the step cap, left no
answer.pyand scored zero, and those fiveaccounted for 3.135 of a 3.138 total gap — 99.9%. But
step_limit: 0ismini-swe-agent's own default and 100 was ours, and those five stopped with
36–52% of the cost budget unspent. Re-run on mini's default they all submitted,
none using more than 60 steps, scoring 0.316 to 0.9996. Those zeros were
single-run variance, not a budget. The table above already uses the corrected
values.
An ablation on the image channel
The render reaches the model twice. Its own code reads the array and reduces it
to a number, and the picture is also attached to the next turn. Only the second
was removed (
BENCH_NO_IMAGE_FEEDBACK, default off), so the model can stillrender and still compare; it just cannot look.
An earlier revision of this description reported the arm on 16 records, where
both intervals excluded zero, and read that as the picture steering the search.
The 100-record run does not support that reading:
Without the picture the model spends a few more rounds and lands in the same
place. That agrees with mini-swe-agent, which never receives a render and ties
with this harness on the same task (below).
View-information arms
BENCH_VIEW_HINTswaps only the camera block of the system prompt. Unset, theoutput is byte-identical to before, so the 299-record baseline is untouched.
anchorstates only the top-right camera, andperturbis for targets whoseother three views were re-rendered off the corner directions. The block is
load-bearing: 86% of baseline programs transcribe those vectors into their own
code.
Limitations
that matters: identical configuration, identical record, 0.000 one time and
0.9996 the next. Every interval here covers sampling across records, not
repeats of one.
Didn't receive a 'response.completed' eventfrom the provider, each after 3–4 attempts. Theobvious explanation is wrong: their prompt-token medians are 42% of the
successful records', so it is not context size.
censors the distribution it would need to be sized against, so the right
value is unknown.
image files, standard Python libraries and an image cropping tool. This also
installs CadQuery and ships the renderer that made the targets.