feat(openai): optional vision gating to strip images for non-vision models - #257
Open
JumpLink wants to merge 1 commit into
Open
feat(openai): optional vision gating to strip images for non-vision models#257JumpLink wants to merge 1 commit into
JumpLink wants to merge 1 commit into
Conversation
JumpLink
marked this pull request as ready for review
June 20, 2026 06:23
This was referenced Jun 20, 2026
JumpLink
force-pushed
the
feat/vision-capability
branch
from
July 28, 2026 15:49
f295fc3 to
2b9bb71
Compare
JumpLink
force-pushed
the
feat/vision-capability
branch
from
July 31, 2026 12:33
2b9bb71 to
964f09c
Compare
JumpLink
added a commit
to faktenforum/LibreChat
that referenced
this pull request
Aug 14, 2026
Add a `vision` option to getOpenAIConfig / getOpenAILLMConfig that is
forwarded onto the OpenAI llmConfig (`OAIClientOptions.vision`). When a
caller knows the target model has no vision support, it can set
`vision: false` so the chat client strips image content before sending,
avoiding hard provider errors ("model is not a multimodal model" / "No
endpoints found that support image input").
Defaults to undefined, so existing behavior is unchanged. The image
stripping itself is implemented in @librechat/agents (see
danny-avila/agents#257), which this option drives.
Covered by getOpenAILLMConfig unit tests (vision true/false/absent).
…odels Refs danny-avila/LibreChat#11418, danny-avila/LibreChat#14341 When image content reaches a model without vision support, OpenAI-compatible providers reject the entire request: - `model is not a multimodal model` (Scaleway/Mistral) - `No endpoints found that support image input` (OpenRouter) The caller often cannot prevent the image from being in the history: - a tool returns an image (an image-generation tool feeding its artifact back), - or an agent hands off from a vision-capable model to a text-only one, carrying the user's original `image_url` block along - reported independently in LibreChat#14341. There is currently no way to tell the client "this model cannot take images". An opt-in `vision` flag on the OpenAI-family chat classes. When `vision: false`, image parts are stripped at a single choke point, immediately before the message stream is delegated to the base class - so it does not depend on the internal message-conversion path. `stripImagesFromMessages(messages, visionCapable)`: - returns the input unchanged when `visionCapable` is true (the default), so existing behaviour is untouched; - clones only the messages that actually carry an image; - keeps a short text placeholder when stripping would leave a message empty; - covers **both** OpenAI-style `image_url` parts and standard `image` data content blocks. The latter matters: they are converted to `image_url` further downstream, so dropping only `image_url` still lets an uploaded image through to a text-only model. Wired into `ChatOpenAI`, `AzureChatOpenAI`, `ChatDeepSeek` and `ChatXAI`. `ChatOpenAI._streamRawResponseChunks` is covered too, which is the path `ChatOpenRouter` streams through. ```ts new ChatOpenAI({ /* ...existing fields... */, vision: false }); ``` Defaults to `true` everywhere - non-breaking. `src/llm/openai/utils/stripImages.test.ts`: 7 cases over `image_url` parts, data content blocks, the placeholder substitution, and the vision-capable pass-through. Full `src/llm/openai` + `src/messages` suites pass (752), tsc and eslint clean. Signed-off-by: JumpLink <pascal@artandcode.studio>
JumpLink
force-pushed
the
feat/vision-capability
branch
from
August 14, 2026 09:47
964f09c to
e23bf0e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(openai): optional vision gating to strip images for non-vision models
Refs danny-avila/LibreChat#11418, danny-avila/LibreChat#14341
Problem
When image content reaches a model without vision support, OpenAI-compatible
providers reject the entire request:
model is not a multimodal model(Scaleway/Mistral)No endpoints found that support image input(OpenRouter)The caller often cannot prevent the image from being in the history:
the user's original
image_urlblock along - reported independently inLibreChat#14341.
There is currently no way to tell the client "this model cannot take images".
Approach
An opt-in
visionflag on the OpenAI-family chat classes. Whenvision: false,image parts are stripped at a single choke point, immediately before the message
stream is delegated to the base class - so it does not depend on the internal
message-conversion path.
stripImagesFromMessages(messages, visionCapable):visionCapableis true (the default), soexisting behaviour is untouched;
image_urlparts and standardimagedatacontent blocks. The latter matters: they are converted to
image_urlfurtherdownstream, so dropping only
image_urlstill lets an uploaded image throughto a text-only model.
Wired into
ChatOpenAI,AzureChatOpenAI,ChatDeepSeekandChatXAI.ChatOpenAI._streamRawResponseChunksis covered too, which is the pathChatOpenRouterstreams through.API
Defaults to
trueeverywhere - non-breaking.Tests
src/llm/openai/utils/stripImages.test.ts: 7 cases overimage_urlparts, datacontent blocks, the placeholder substitution, and the vision-capable pass-through.
Full
src/llm/openai+src/messagessuites pass (623), tsc and eslint clean.Signed-off-by: JumpLink pascal@artandcode.studio