Repository navigation
fix(agent): allow tool responses to pass image content to next-round vision input - #7913
Closed
lirenjie95 wants to merge 1 commit into
Closed
lirenjie95 wants to merge 1 commit into
lirenjie95 wants to merge 1 commit into
Conversation
Collaborator
|
需要修改 llm request 的 tool content 类型,同时修改所有 tool response,支持返回符合类型。参考 openai completions 接口文档。 |
5 of 7 tasks
Author
|
已按维护者意见完整实现:工具全链路支持主动返回 content parts(text / image_url),字符串响应仅在整串恰好为图片链接时兜底转图片。已重新提交为新 PR #7923,此 PR 保持关闭。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7827
背景
Agent 工具(如沙盒 get_file_url、workflow 工具节点)返回图片后,下一轮模型请求无法把图片作为视觉输入,只能拿到文本链接。
改动内容
对齐 OpenAI chat completions 的 tool message content 规范,允许工具消息返回 text / image_url content parts,并让整条工具链路支持主动产出图片:
ChatCompletionToolMessageParamSchema.content支持字符串或ChatCompletionToolMessageContentPart[](text / image_url)AgentLoopToolExecutionResult/AgentLoopCoreToolRunResult/ sandbox 工具结果新增可选content字段,逐层透传{ content: [{ type: 'image_url', image_url: { url } }] }时提取为 tool content;普通文本输出行为不变get_file_url对图片文件在文本链接之外额外返回 image_url parts(工具主动返回 content 的示例)测试
兼容性