[silencelamb]

Motivation

Parallel tool results can return images in a different order from the assistant's tool calls. LMDeploy currently loses their ownership in two places:

• The Anthropic adapter splits a result into a tool text message and a separate user image message, removing the image's direct association with tool_use_id.
• The GLM-5.3-Flash template renders unambiguous tool-result groups in call order, while multimodal preprocessing collects images in message order. Reversed results can therefore associate an image with the wrong tool even through native Chat.

Responses additionally rejects images inside function_call_output and emits consecutive parallel calls as separate assistant messages.

Modification

• Keep Anthropic text and images together in a tool message with its tool_call_id, preserving internal block order. Flush ordinary user content before each tool result so surrounding text and images keep their original position.
• Add a chat-template preparation hook used by the shared multimodal path before prompt rendering and media collection. Apply tool-result ordering only to the GLM-5.3-Flash HF template, only for contiguous multimodal result groups immediately following assistant calls with unambiguous IDs. Other templates, text-only groups, and ambiguous groups retain their original order; no sorting crosses user-message or conversation-round boundaries.
• Support Responses function_call_output.output text/image arrays through the existing image converter, and group consecutive function_call items into one assistant call group.
• Add protocol conversion, HTTP/SSE, real-image-decoding, and shared preprocessing regressions, and update English/Chinese documentation.

The model-specific gate matches Glm5NextForConditionalGeneration, the multimodal architecture used by GLM-5.3-Flash. Plain GLM-5.3 uses GlmMoeDsaForCausalLM and does not enter this ordering path.

BC-breaking (Optional)

No new dependency requirements or public request fields. Tool images now stay attached to their tool result instead of becoming standalone user messages. Text-only tool-result formatting is preserved; consecutive Responses calls are correctly grouped as one parallel invocation.

Use cases (Optional)

Replay screenshot tools whose results arrive in reverse order, or return multiple images and surrounding text from each tool. Chat, Messages, and Responses use the same multimodal preparation path. Responses examples are included in the updated documentation.

Validation

• Complete serving test suite: 685 passed (python -m pytest tests/test_lmdeploy/serve -q). HTTP/SSE tests use fake generation; shared preprocessing tests decode real image data.
• Actual GLM-5.3-Flash tokenizer, chat template, and CPU vision preprocessor: 36/36 combinations matched their same-protocol, call-order controls, comparing complete input IDs, visual tensors, and offsets. The matrix covers Chat/Messages/Responses, all six permutations of three tools, and one/two images per tool.
• Three additional real-template checks confirmed that missing, duplicate, and unknown result IDs preserve input order for both rendered text and collected images.
• Before this change, the same 36-case CPU matrix produced 20 ordering mismatches, 12 rejected Responses image-output cases, and four matching ordered controls.
• Applicable pre-commit hooks and git diff --check passed.
• No GPU service was started. Full GLM generation and Router integration validation are deferred. CPU preprocessing validation does not establish final model-answer accuracy.

The PR1 base now also includes 150 shared user-image regressions; they pass together with the PR2 tool-image tests.

Three-endpoint coverage

Layer	/v1/chat/completions	/v1/messages	/v1/responses
Protocol conversion/validation and real image decoding	Covered	Covered	Covered
Shared preprocessing: six tool-result permutations, one/two images per tool	Covered	Covered	Covered
HTTP non-streaming and SSE, with fake generation	Covered	Covered	Covered
Actual GLM-5.3-Flash CPU preprocessing	Covered	Covered	Covered
Full GLM-5.3-Flash GPU generation	Deferred	Deferred	Deferred

tests/test_lmdeploy/serve/test_tool_images.py parameterizes protocol over chat, messages, and responses. Its main matrix has 36 cases (3 protocols × 6 permutations × 2 image counts). These are conversion/shared-preprocessing tests, not HTTP requests. Additional tests cover ambiguous IDs, partial groups, mixed text/image results, non-GLM templates, and user/round boundaries.

HTTP/SSE regression locations:

• Chat: tests/test_lmdeploy/serve/openai/chat_completions/test_chat_tool_images.py.
• Messages: tests/test_lmdeploy/serve/anthropic/test_endpoints.py::test_http_tool_result_images_keep_their_ids_and_content_order.
• Responses: tests/test_lmdeploy/serve/openai/responses/test_images.py::test_http_tool_images_preserve_parallel_calls_and_result_ownership.

Dependency and review scope

This is PR2, based on the Responses user-image commit in InternLM/lmdeploy#5043. This fork PR targets feat/responses-image-input so its diff contains only the tool-image changes. Submit the incremental fix upstream after PR1 merges.

Checklist

• Pre-commit checks passed.
• Regression tests added; serving suite passed.
• No new downstream dependency/version requirements.
• English and Chinese documentation updated.