Skip to content

serve: start an elastic image encoder before encoding a tool-returned image (#3171) - #3174

Open
Lithivm wants to merge 1 commit into
Niko1221:mainfrom
Lithivm:fix/elastic-vision-tool-image
Open

Lithivm wants to merge 1 commit into
Niko1221:mainfrom
Lithivm:fix/elastic-vision-tool-image

Conversation

@Lithivm

@Lithivm Lithivm commented Oct 19, 2026

Copy link
Copy Markdown

Fixes #3171.

The bug

_note_unreadable_tool_images() encodes a picture that a tool returned (Claude Code's Read of an image file, an MCP tool returning a plot) with self.vision.encode(src) — but unlike prepare()'s own image path it never calls self._vision_up():

# prepare(), the request's own images — correct
with self._turn(cancel):
    self._vision_up()
    encoded = self.vision.encode_all(images, cancel) ...

# _note_unreadable_tool_images(), a picture a tool returned — no _vision_up()
with self._turn(cancel):  # the encoder takes its turn with the requests (see below)
    self.vision.encode(src)

With the elastic encoder (vram_elastic: true + vision "elastic": true) the encoder is not running between pictures, so Vision.proc is None and encode()'s self.proc.stdin.write(...) raises AttributeError. That function's except (ValueError, OSError) does not catch it, so it leaves prepare() and lands in _post's generic handler, which reports any of those as a client error:

400 {"type": "invalid_request_error",
     "message": "the request is malformed (AttributeError: 'NoneType' object has no attribute 'stdin')"}

So an agent doing a normal tool call breaks the turn — and, because clients re-send history, possibly the turns after it as well.

Repro (0.1.44.1, no waiting needed)

A fresh elastic session never starts the encoder (it starts with the first picture), so the first request with a tool image fails:

  1. "vram_elastic": true and "vision": {"gpu": true, "elastic": true, ...}; start the server — strata-vision is not running (expected).
  2. One request whose messages hold an image in a tool message (Anthropic dialect: user → assistant tool_use → user tool_result with an image block).
  3. 400 as above. Measured: fails before this patch, answers normally after it, and the encoder is then started and stopped as usual (VRAM 3584 MiB kept free: the expert cache N -> M of N slots in the engine log).

"elastic": false does not hit it (the encoder runs from the start), and a user-sent image was always fine — only the tool-returned path is affected.

The change

Two lines, the same call the sibling path makes. _vision_up() returns immediately when the encoder is not elastic or is already running (if not getattr(v, "elastic", False) or v.alive(): return), so nothing changes for non-elastic setups.

                         with self._turn(cancel):  # the encoder takes its turn with the requests (see below)
+                            self._vision_up()   # an elastic encoder is down between pictures: start it first, as
+                                                # prepare()'s own image path does (else encode() writes to a None proc)
                             self.vision.encode(src)

\ No newline at end of file — no; the file's LF endings are unchanged (2 lines added, 0 removed).

Two things I did not change (see #3171)

  • That function exists so an unreadable tool picture becomes [image omitted: ...] instead of a 400; widening its except to AttributeError would keep that promise for unexpected failures too. Kept out of this PR to keep it to the defect.
  • _post maps any AttributeError raised while preparing a request to 400 invalid_request_error. That is why this looked like a client bug; an internal error arguably deserves a 500.

Verified

  • Before: 400 with the exact message above (0.1.44.1, Windows 11 26200, RTX 5070 Ti, Q2_0, context 131072, --mmap-experts, --vram-elastic, elastic_mib 3584).
  • After: the same request answers; regression checks pass on plain text (/v1/chat/completions), a user image via /v1/chat/completions and via /v1/messages, and the tool image that used to fail.

… image

_note_unreadable_tool_images() encodes a picture a tool returned (Claude Code's
Read of an image file, an MCP tool returning a plot) with self.vision.encode(),
but unlike prepare()'s own image path it never calls self._vision_up().  With the
elastic encoder (vram_elastic + vision "elastic") the encoder is not running
between pictures, so Vision.proc is None and encode()'s `self.proc.stdin.write`
raises AttributeError.  The request then fails:

    400 {"type": "invalid_request_error",
         "message": "the request is malformed (AttributeError: 'NoneType' object
                     has no attribute 'stdin')"}

That function's own except (ValueError, OSError) does not catch it, so the
exception escapes prepare() and is reported as a client error.

Reproduced on 0.1.44.1 with a fresh elastic session (the encoder never started,
so no waiting for elastic_idle_s is needed): one /v1/messages request with a
tool_result image gives the 400 above; the same request succeeds with
"elastic": false.  Fixes Niko1221#3171.

_vision_up() is a no-op when the encoder is not elastic or is already running, so
non-elastic setups are unaffected.
@ogrrd

ogrrd commented Oct 21, 2026

Copy link
Copy Markdown

Confirmed on real hardware, and still broken on current main (checked today at v0.1.46.3: _note_unreadable_tool_images still encodes without a wake, and the neighbouring except (ValueError, OSError) lets the AttributeError sail through to a 400).

  • Windows 11, RTX 5090, elastic encoder ("vram_elastic": true + vision "elastic": true), installed v0.1.44.1.
  • The trigger is ordinary OpenAI-dialect agent traffic, not just the Anthropic shape in [Bug]: elastic vision — a tool-returned image crashes the request (400 "malformed", AttributeError on a None encoder proc) #3171: our agent runtime (Hermes) injects its vision-tool results as image_url parts on tool messages via /v1/chat/completions. Every turn carrying that history answered 400 the request is malformed (AttributeError: 'NoneType' object has no attribute 'stdin') — and since clients resend history, one picture poisons the whole conversation.
  • Ran this patch (plus a self.last_image_at = time.time() after the encode) since today, on the box's everyday inference: the encoder wakes elastically on the tool picture, the pads expand, no more 400s.

Two notes for whoever reviews, both minor:

  1. _vision_up() itself doesn't touch last_image_at, so on the success path the timer is refreshed a moment later by prepare()'s pad-expansion loop — which means the one-liner is functionally complete, but if encode() raises (a picture the encoder refuses), the freshly-started encoder sits behind a stale timer. Updating last_image_at around the turn, as the closed [Other] serve: a tool's picture starts the elastic encoder, and its turn counts #2641 did in a finally, costs nothing and closes that window.
  2. Worth pulling [Other] serve: a tool's picture starts the elastic encoder, and its turn counts #2641's other half too: if _vision_up()'s restart raises RuntimeError (no VRAM, encoder binary fails), that also escapes the except (ValueError, OSError) and 400s the request — the method's whole contract is "become a note, not a dead request", so RuntimeError belongs in that filter.

+1 from a Windows/5090 box where this was the single most-hit bug of the week — glad to retest any revision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: elastic vision — a tool-returned image crashes the request (400 "malformed", AttributeError on a None encoder proc)

2 participants