Skip to content

feat(openai): optional vision gating to strip images for non-vision models - #257

Open
JumpLink wants to merge 1 commit into
danny-avila:mainfrom
faktenforum:feat/vision-capability
Open

feat(openai): optional vision gating to strip images for non-vision models#257
JumpLink wants to merge 1 commit into
danny-avila:mainfrom
faktenforum:feat/vision-capability

Conversation

@JumpLink

@JumpLink JumpLink commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

feat(openai): optional vision gating to strip images for non-vision models

Refs danny-avila/LibreChat#11418, danny-avila/LibreChat#14341

Problem

When image content reaches a model without vision support, OpenAI-compatible
providers reject the entire request:

  • model is not a multimodal model (Scaleway/Mistral)
  • No endpoints found that support image input (OpenRouter)

The caller often cannot prevent the image from being in the history:

  • a tool returns an image (an image-generation tool feeding its artifact back),
  • or an agent hands off from a vision-capable model to a text-only one, carrying
    the user's original image_url block along - reported independently in
    LibreChat#14341.

There is currently no way to tell the client "this model cannot take images".

Approach

An opt-in vision flag on the OpenAI-family chat classes. When vision: false,
image parts are stripped at a single choke point, immediately before the message
stream is delegated to the base class - so it does not depend on the internal
message-conversion path.

stripImagesFromMessages(messages, visionCapable):

  • returns the input unchanged when visionCapable is true (the default), so
    existing behaviour is untouched;
  • clones only the messages that actually carry an image;
  • keeps a short text placeholder when stripping would leave a message empty;
  • covers both OpenAI-style image_url parts and standard image data
    content blocks. The latter matters: they are converted to image_url further
    downstream, so dropping only image_url still lets an uploaded image through
    to a text-only model.

Wired into ChatOpenAI, AzureChatOpenAI, ChatDeepSeek and ChatXAI.
ChatOpenAI._streamRawResponseChunks is covered too, which is the path
ChatOpenRouter streams through.

API

new ChatOpenAI({ /* ...existing fields... */, vision: false });

Defaults to true everywhere - non-breaking.

Tests

src/llm/openai/utils/stripImages.test.ts: 7 cases over image_url parts, data
content blocks, the placeholder substitution, and the vision-capable pass-through.
Full src/llm/openai + src/messages suites pass (623), tsc and eslint clean.

Signed-off-by: JumpLink pascal@artandcode.studio

@JumpLink
JumpLink marked this pull request as ready for review June 20, 2026 06:23
@JumpLink
JumpLink force-pushed the feat/vision-capability branch from f295fc3 to 2b9bb71 Compare July 28, 2026 15:49
@JumpLink
JumpLink force-pushed the feat/vision-capability branch from 2b9bb71 to 964f09c Compare July 31, 2026 12:33
JumpLink added a commit to faktenforum/LibreChat that referenced this pull request Aug 14, 2026
Add a `vision` option to getOpenAIConfig / getOpenAILLMConfig that is
forwarded onto the OpenAI llmConfig (`OAIClientOptions.vision`). When a
caller knows the target model has no vision support, it can set
`vision: false` so the chat client strips image content before sending,
avoiding hard provider errors ("model is not a multimodal model" / "No
endpoints found that support image input").

Defaults to undefined, so existing behavior is unchanged. The image
stripping itself is implemented in @librechat/agents (see
danny-avila/agents#257), which this option drives.

Covered by getOpenAILLMConfig unit tests (vision true/false/absent).
…odels

Refs danny-avila/LibreChat#11418, danny-avila/LibreChat#14341

When image content reaches a model without vision support, OpenAI-compatible
providers reject the entire request:

- `model is not a multimodal model` (Scaleway/Mistral)
- `No endpoints found that support image input` (OpenRouter)

The caller often cannot prevent the image from being in the history:

- a tool returns an image (an image-generation tool feeding its artifact back),
- or an agent hands off from a vision-capable model to a text-only one, carrying
  the user's original `image_url` block along - reported independently in
  LibreChat#14341.

There is currently no way to tell the client "this model cannot take images".

An opt-in `vision` flag on the OpenAI-family chat classes. When `vision: false`,
image parts are stripped at a single choke point, immediately before the message
stream is delegated to the base class - so it does not depend on the internal
message-conversion path.

`stripImagesFromMessages(messages, visionCapable)`:

- returns the input unchanged when `visionCapable` is true (the default), so
  existing behaviour is untouched;
- clones only the messages that actually carry an image;
- keeps a short text placeholder when stripping would leave a message empty;
- covers **both** OpenAI-style `image_url` parts and standard `image` data
  content blocks. The latter matters: they are converted to `image_url` further
  downstream, so dropping only `image_url` still lets an uploaded image through
  to a text-only model.

Wired into `ChatOpenAI`, `AzureChatOpenAI`, `ChatDeepSeek` and `ChatXAI`.
`ChatOpenAI._streamRawResponseChunks` is covered too, which is the path
`ChatOpenRouter` streams through.

```ts
new ChatOpenAI({ /* ...existing fields... */, vision: false });
```

Defaults to `true` everywhere - non-breaking.

`src/llm/openai/utils/stripImages.test.ts`: 7 cases over `image_url` parts, data
content blocks, the placeholder substitution, and the vision-capable pass-through.
Full `src/llm/openai` + `src/messages` suites pass (752), tsc and eslint clean.

Signed-off-by: JumpLink <pascal@artandcode.studio>
@JumpLink
JumpLink force-pushed the feat/vision-capability branch from 964f09c to e23bf0e Compare August 14, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant