[Fix] Preserve tool-result image ownership

docs/en/llm/api_server_anthropic.md

@@ -18,6 +18,7 @@ For Anthropic `POST` endpoints, include:
 ## Notes and Current Limits
 
 - `POST /v1/messages` supports text output, reasoning output (`thinking` blocks), and tool-use output (`tool_use` blocks).
+- With a vision-capable model, `tool_result.content` can contain text and image blocks. These blocks remain in the same tool result, retaining `tool_use_id` and their internal order. Ordinary user content before, between, or after tool results retains its position.
 - Tool use requires launching API server with a configured tool parser (`--tool-call-parser ...`).
 - Reasoning block extraction depends on parser configuration (same parser stack used by OpenAI-compatible chat endpoint).
 - `count_tokens` is tokenizer/chat-template based and is intended for practical estimation.

docs/en/llm/api_server_responses.md

@@ -16,11 +16,12 @@ For `POST /v1/responses`, include:
 
 ## Notes and Current Limits
 
-- `POST /v1/responses` supports text and user-message image inputs with a vision-capable model.
+- `POST /v1/responses` supports text, user-message images, and tool-result images with a vision-capable model.
 - `input` may be a string or a list of Responses input items.
 - User content arrays can mix `input_text` and `input_image` blocks. Images use `image_url` with an HTTP(S) URL or a base64 image data URL. Multiple images and their order are preserved in both streaming and non-streaming requests.
 - Image `detail` and the `min_dynamic_patch` / `max_dynamic_patch` extensions are forwarded like Chat image parameters; their effect depends on the model's image processor.
-- Image `file_id` references, images in system/developer/assistant messages, and images in `function_call_output` are not supported.
+- `function_call_output.output` can also contain an array of `input_text` and `input_image` blocks. Images stay associated with the result's `call_id`. Consecutive `function_call` items are treated as one parallel tool-call group.
+- Image `file_id` references and images in system/developer/assistant messages are not supported.
 - `instructions`, `developer`, and `system` messages are merged into a single leading system message for chat-template compatibility.
 - Function tools are converted to LMDeploy's OpenAI-compatible tool format. Tool calling requires launching API server with a configured tool parser (`--tool-call-parser ...`).
 - `parallel_tool_calls` defaults to `true`. When it is `false`, LMDeploy follows vLLM-style compatibility and returns only the first parsed function call.
@@ -122,3 +123,25 @@ When `stream=true`, the endpoint returns `text/event-stream` events such as:
 ## Codex Integration
 
 May refer to [codex](../integration/codex.md).
+
+## Example: tool results with images
+
+Replay the tool call and return its image using the same `call_id`. For parallel calls, place the consecutive `function_call` items before their results; results may arrive in a different order. Each result preserves its internal text/image order.
+
+```json
+{
+  "model": "your-vision-model",
+  "input": [
+    {"role": "user", "content": "Describe the screenshot."},
+    {"type": "function_call", "call_id": "call_screenshot", "name": "screenshot", "arguments": "{}"},
+    {
+      "type": "function_call_output",
+      "call_id": "call_screenshot",
+      "output": [
+        {"type": "input_text", "text": "Screenshot result:"},
+        {"type": "input_image", "image_url": "https://example.com/screenshot.png"}
+      ]
+    }
+  ]
+}
+```

docs/zh_cn/llm/api_server_anthropic.md

@@ -18,6 +18,7 @@ LMDeploy 提供了轻量级的 Anthropic 兼容接口，方便接入 Anthropic 
 ## 说明与当前限制
 
 - `POST /v1/messages` 支持文本输出、推理输出（`thinking` block）和工具调用输出（`tool_use` block）。
+- 使用视觉模型时，`tool_result.content` 可以包含 text 和 image 块。这些块保留在同一工具结果中，并保留 `tool_use_id` 和内部顺序；工具结果前、后及结果之间的普通 user 内容也保留原有位置。
 - 工具调用需要在启动 API Server 时配置工具解析器（`--tool-call-parser ...`）。
 - 推理内容 block 的提取依赖解析器配置（与 OpenAI 兼容 chat 接口使用同一套解析器）。
 - `count_tokens` 基于 tokenizer 和 chat template 计算，适合用于实用估算。

docs/zh_cn/llm/api_server_responses.md

@@ -16,11 +16,12 @@ LMDeploy 提供了轻量级的 OpenAI Responses 兼容接口，方便接入使
 
 ## 说明与当前限制
 
-- `POST /v1/responses` 支持文本输入，以及视觉模型的 user 消息图片输入。
+- `POST /v1/responses` 支持文本输入，以及视觉模型的 user 消息图片和工具结果图片输入。
 - `input` 可以是字符串，也可以是 Responses input item 列表。
 - user 的 content 数组可以混合 `input_text` 和 `input_image`。图片通过 `image_url` 指定 HTTP(S) URL 或 base64 图片 data URL；流式和非流式请求均支持多图，并保留图文顺序。
 - 图片的 `detail` 和 `min_dynamic_patch` / `max_dynamic_patch` 扩展参数按 Chat 图片参数方式传递，实际效果取决于模型的图片处理器。
-- 暂不支持图片 `file_id` 引用、system/developer/assistant 消息中的图片，以及 `function_call_output` 中的图片。
+- `function_call_output.output` 也可以包含 `input_text` 和 `input_image` 数组。图片保留所属结果的 `call_id`，连续的 `function_call` 条目会转换为同一组并行工具调用。
+- 暂不支持图片 `file_id` 引用，以及 system/developer/assistant 消息中的图片。
 - `instructions`、`developer` 和 `system` 消息会合并为一个位于最前面的 system message，以兼容 chat template。
 - function tools 会转换为 LMDeploy 的 OpenAI 兼容 tool 格式。工具调用需要在启动 API Server 时配置工具解析器（`--tool-call-parser ...`）。
 - `parallel_tool_calls` 默认为 `true`。当设置为 `false` 时，LMDeploy 会保持与 vLLM 类似的兼容行为，仅返回解析到的第一个 function call。
@@ -137,3 +138,25 @@ stream_idle_timeout_ms = 300000
 ```
 
 `model` 的值必须与 LMDeploy 暴露的模型名完全一致。
+
+## 示例：包含图片的工具结果
+
+重放工具调用，并使用相同的 `call_id` 返回图片。并行调用时，将连续的 `function_call` 条目放在结果之前；结果可以按不同顺序返回，每个结果内部的图文顺序都会保留。
+
+```json
+{
+  "model": "your-vision-model",
+  "input": [
+    {"role": "user", "content": "Describe the screenshot."},
+    {"type": "function_call", "call_id": "call_screenshot", "name": "screenshot", "arguments": "{}"},
+    {
+      "type": "function_call_output",
+      "call_id": "call_screenshot",
+      "output": [
+        {"type": "input_text", "text": "Screenshot result:"},
+        {"type": "input_image", "image_url": "https://example.com/screenshot.png"}
+      ]
+    }
+  ]
+}
+```

lmdeploy/model.py

@@ -114,6 +114,10 @@ def from_json(cls, file_or_string):
 class BaseChatTemplate:
     """Base Chat template."""
 
+    def prepare_multimodal_messages(self, messages: list[dict]) -> list[dict]:
+        """Prepare message order before rendering and collecting media."""
+        return messages
+
     def __init__(self,
                  system='',
                  meta_instruction='',
@@ -788,11 +792,50 @@ def __init__(self, model_path: str = '', trust_remote_code: bool = False, **kwar
                 self.stop_words.append(self.tokenizer.eot_token)
             arch, _ = get_model_arch(model_path)
             self.is_gpt_oss = arch == 'GptOssForCausalLM'
+            # GLM-5.3-Flash's multimodal architecture, not the text-only GLM-5.3.
+            self.is_glm5_next = arch == 'Glm5NextForConditionalGeneration'
             if self.is_gpt_oss:
                 self.stop_words.append('<|call|>')
         except Exception as e:
             raise ValueError(f'Try apply_chat_template failed: {e}')
 
+    def prepare_multimodal_messages(self, messages: list[dict]) -> list[dict]:
+        """Match GLM-5.3-Flash tool rendering order before collecting media.
+
+        Its template renders an unambiguous, contiguous result group in the preceding assistant's tool-call order. Both
+        the prompt and media collector must see that order. Other templates keep their original message order.
+        """
+        if not self.is_glm5_next:
+            return messages
+        ordered = list(messages)
+        start = 0
+        while start < len(ordered):
+            if ordered[start]['role'] != 'tool':
+                start += 1
+                continue
+            end = start + 1
+            while end < len(ordered) and ordered[end]['role'] == 'tool':
+                end += 1
+            group = ordered[start:end]
+            has_media = any(isinstance(message.get('content'), list)
+                            and any(isinstance(part, dict) and part.get('type') in ('image', 'video', 'audio')
+                                    for part in message['content']) for message in group)
+            previous = ordered[start - 1] if start else {}
+            calls = (previous.get('tool_calls') or []) if previous.get('role') == 'assistant' else []
+            # Match the template's tool_call_id/id fallback. Ambiguous groups
+            # are rendered in input order, so leave their media in that order.
+            call_ids = [call.get('tool_call_id') or call.get('id') for call in calls]
+            result_ids = [message.get('tool_call_id') or message.get('id') for message in group]
+            valid_ids = all(isinstance(identifier, str) and identifier for identifier in call_ids + result_ids)
+            if (has_media and call_ids and valid_ids
+                    and len(set(call_ids)) == len(call_ids) and len(set(result_ids)) == len(result_ids)
+                    and set(result_ids).issubset(call_ids)):
+                positions = {identifier: index for index, identifier in enumerate(call_ids)}
+                ordered[start:end] = sorted(group, key=lambda message: positions[message.get('tool_call_id')
+                                                                                or message.get('id')])
+            start = end
+        return ordered
+
     def get_prompt(self, prompt, **kwargs):
         messages = [{'role': 'user', 'content': prompt}]
         return self.messages2prompt(messages, **kwargs)

lmdeploy/serve/anthropic/adapter.py

@@ -251,37 +251,28 @@ def _merge_system_messages(messages: list[_MessageT]) -> list[_MessageT]:
 
 
 def _convert_user_tool_result(block: ContentBlockParam | dict[str, Any]) -> list[dict[str, Any]]:
-    tool_text = ''
-    tool_image_urls: list[str] = []
     content = _block_get(block, 'content')
-
+    tool_content: str | list[dict[str, Any]] = ''
     if isinstance(content, str):
-        tool_text = content
+        tool_content = content
     elif isinstance(content, list):
-        text_parts: list[str] = []
+        parts: list[dict[str, Any]] = []
         for item in content:
             if _block_get(item, 'type') == 'text':
-                text_parts.append(_block_get(item, 'text', ''))
+                parts.append(dict(type='text', text=_block_get(item, 'text', '')))
             elif _block_get(item, 'type') == 'image':
                 url = _convert_image_source_to_url(_block_get(item, 'source', {}))
                 if url:
-                    tool_image_urls.append(url)
-        tool_text = '\n'.join(text_parts)
-
-    messages = [
-        dict(
-            role='tool',
-            tool_call_id=_block_get(block, 'tool_use_id') or _block_get(block, 'id') or '',
-            content=tool_text or '',
-        )
-    ]
-    if tool_image_urls:
-        messages.append(
-            dict(
-                role='user',
-                content=[dict(type='image_url', image_url=dict(url=url)) for url in tool_image_urls],
-            ))
-    return messages
+                    parts.append(dict(type='image_url', image_url=dict(url=url)))
+        # A separate user image message loses its tool_call_id and interrupts
+        # the contiguous tool-result group used by chat templates.
+        if any(part['type'] == 'image_url' for part in parts):
+            tool_content = parts
+        else:
+            tool_content = '\n'.join(part['text'] for part in parts)
+    return [dict(role='tool',
+                 tool_call_id=_block_get(block, 'tool_use_id') or _block_get(block, 'id') or '',
+                 content=tool_content)]
 
 
 def to_openai_messages(
@@ -340,6 +331,13 @@ def to_openai_messages(
 
             if block_type == 'tool_result':
                 if message.role == 'user':
+                    # Ordinary user content is a boundary between tool groups.
+                    if content_parts:
+                        content = (content_parts[0]['text']
+                                   if len(content_parts) == 1 and content_parts[0]['type'] == 'text'
+                                   else content_parts)
+                        openai_messages.append(dict(role='user', content=content))
+                        content_parts = []
                     openai_messages.extend(_convert_user_tool_result(block))
                 else:
                     result_text = _text_from_block_content(

lmdeploy/serve/openai/responses/request.py

@@ -278,12 +278,15 @@ def messages_from_input(request: ResponsesRequest) -> list[dict[str, Any]]:
         messages.append(dict(role='user', content=request.input))
         return ([dict(role='system', content='\n\n'.join(system_parts))] if system_parts else []) + messages
 
+    previous_item_type = None
     for idx, item in enumerate(request.input):
         item = _as_dict(item)
         if not item:
             raise ValueError(f'Unsupported Responses input item at index {idx}.')
 
         item_type = item.get('type', 'message')
+        adjacent_call = previous_item_type == 'function_call'
+        previous_item_type = item_type
         if item_type == 'function_call':
             call_id = item.get('call_id') or item.get('id')
             name = item.get('name')
@@ -292,21 +295,13 @@ def messages_from_input(request: ResponsesRequest) -> list[dict[str, Any]]:
             arguments = item.get('arguments') or ''
             if not isinstance(arguments, str):
                 raise ValueError(f'Unsupported `arguments` in function_call item at index {idx}. Expected string.')
-            messages.append(
-                dict(
-                    role='assistant',
-                    content=None,
-                    tool_calls=[
-                        dict(
-                            id=call_id,
-                            type='function',
-                            function=dict(
-                                name=name,
-                                arguments=arguments,
-                            ),
-                        )
-                    ],
-                ))
+            call = dict(id=call_id, type='function', function=dict(name=name, arguments=arguments))
+            # Adjacent calls belong to one parallel invocation. Keep all IDs
+            # visible to templates associating the following tool results.
+            if adjacent_call:
+                messages[-1]['tool_calls'].append(call)
+            else:
+                messages.append(dict(role='assistant', content=None, tool_calls=[call]))
             continue
         if item_type == 'function_call_output':
             call_id = item.get('call_id')
@@ -316,7 +311,7 @@ def messages_from_input(request: ResponsesRequest) -> list[dict[str, Any]]:
                 dict(
                     role='tool',
                     tool_call_id=call_id,
-                    content=_content_from_input(item.get('output', ''), f'input[{idx}].output'),
+                    content=_content_from_input(item.get('output', ''), f'input[{idx}].output', allow_images=True),
                 ))
             continue
         if item_type != 'message':

lmdeploy/serve/processors/multimodal.py

@@ -426,6 +426,7 @@ async def _get_multimodal_prompt_input(self,
         messages = await self.async_parse_multimodal_item(messages,
                                                           media_io_kwargs,
                                                           allowed_media_domains=self.allowed_media_domains)
+        messages = chat_template.prepare_multimodal_messages(messages)
 
         if self.backend == 'turbomind':
             if self.vl_encoder._uses_new_preprocess:

tests/test_lmdeploy/serve/anthropic/test_adapter_conversion.py

@@ -266,7 +266,7 @@ def test_to_openai_messages_maps_user_tool_result_to_tool_role_message():
     }
 
 
-def test_to_openai_messages_maps_user_tool_result_images_to_user_message():
+def test_to_openai_messages_keeps_tool_result_images_with_tool_id():
     request = _make_request(
         messages=[{
             'role': 'user',
@@ -293,11 +293,10 @@ def test_to_openai_messages_maps_user_tool_result_images_to_user_message():
         {
             'role': 'tool',
             'tool_call_id': 'toolu_123',
-            'content': 'see image',
-        },
-        {
-            'role': 'user',
             'content': [{
+                'type': 'text',
+                'text': 'see image',
+            }, {
                 'type': 'image_url',
                 'image_url': {
                     'url': 'data:image/jpeg;base64,xyz789',

tests/test_lmdeploy/serve/anthropic/test_endpoints.py

@@ -1150,3 +1150,39 @@ def test_messages_rejects_logprobs_when_engine_logprobs_mode_disabled():
     response = _post_messages(client, messages=[{'role': 'user', 'content': 'Hi'}], return_logprob=True)
     assert response.status_code == 400
     assert 'return_logprob' in response.json()['error']['message']
+
+
+@pytest.mark.parametrize('stream', [False, True])
+def test_http_tool_result_images_keep_their_ids_and_content_order(stream):
+    from fastapi import FastAPI
+    from fastapi.testclient import TestClient
+
+    context = _FakeServerContext()
+    app = FastAPI()
+    app.include_router(create_anthropic_router(context))
+    messages = [
+        dict(role='user', content='Read both screenshots.'),
+        dict(role='assistant', content=[dict(type='tool_use', id=identifier, name='screenshot', input={})
+                                        for identifier in ('first', 'second')]),
+        dict(role='user', content=[dict(type='tool_result', tool_use_id=identifier, content=[
+            dict(type='text', text='before'),
+            dict(type='image', source=dict(type='url', url=f'https://example.com/{identifier}.png')),
+            dict(type='text', text='after'),
+        ]) for identifier in ('second', 'first')]),
+    ]
+    with TestClient(app) as client:
+        response = client.post('/v1/messages', headers=ANTHROPIC_HEADERS,
+                               json=_messages_payload(messages=messages, stream=stream))
+    assert response.status_code == 200
+    if stream:
+        assert 'event: message_stop' in response.text
+    else:
+        assert response.json()['stop_reason'] == 'end_turn'
+    forwarded = context.async_engine.preprocess_calls[0][0][0]
+    assert [m['role'] for m in forwarded] == ['user', 'assistant', 'tool', 'tool']
+    for message, identifier in zip(forwarded[2:], ('second', 'first')):
+        assert message['tool_call_id'] == identifier
+        assert message['content'] == [dict(type='text', text='before'),
+                                      dict(type='image_url', image_url=dict(url=f'https://example.com/{identifier}.png')),
+                                      dict(type='text', text='after')]
+    assert len(context.session_mgr.removed) == 1