Skip to content

Support images and PDFs from tools and sub-agents - #92

Merged
lukebuehler merged 2 commits into
mainfrom
media
Sep 12, 2026
Merged

lukebuehler merged 2 commits into
mainfrom
media

Conversation

@lukebuehler

Copy link
Copy Markdown
Contributor

Tools previously left images and PDFs as stored output or rejected them as non-text files, so models could not inspect or reference them. This change delivers admitted media through native provider content and lets models link it in replies with stable media: handles.

Changes

  • Add bounded media admission and companion context entries for native MCP results and read_file on both VFS and execution-environment filesystems. Supported images and PDFs are stored in CAS; unsupported or excess media produces an omission note or tool error.
  • Announce content-derived handles in Anthropic Messages, OpenAI Responses, and OpenAI Chat Completions. Preserve provider-required tool-result ordering and replace tool media with a note for the text-only DeepSeek dialect.
  • Carry media linked in a sub-agent's final answer into its result envelope, retaining the referenced blobs. Runtime activities prepare supplemental context for awaited and joined results; the deterministic engine validates and sequences those entries after the corresponding tool result.
  • Expose media descriptors in the API and generated TypeScript types. Render tool thumbnails, document links, and assistant media: links in the transcript, using browser object URLs for clickable previews with per-view cleanup and loader-scoped fetch caching.
  • Document the capability, provider caveats, and accepted limits; add unit and opt-in live coverage.

Validation

  • cargo test -p engine -p temporal-workflow -p tools -p llm-runtime -p api -p api-projection -p temporal-server --lib --quiet — 1,172 passed, 1 ignored.
  • npm run test --workspace @lightspeed/platform-web -- src/components/session/media.test.tsx src/lib/sessions/transcript.test.ts src/pages/SessionsPage.test.tsx — 85 passed.
  • npm run typecheck --workspace @lightspeed/platform-web — passed.
  • git diff --check origin/main...HEAD — passed.

Live/credentialed suites were not run during PR preparation.

Deployment and accepted limits

The await/join resume activity contract and scheduling change without a Temporal versioning patch. Existing session workflows whose histories already exercised the affected resume paths must be closed before deployment.

The web resolves handles against the loaded transcript; a reference whose originating media is outside that loaded history may remain unresolved. Sub-agent hand-off resolves links against the child's active context, so media compacted out before its final answer is not handed up. A session-wide media lookup is deferred.

@lukebuehler
lukebuehler merged commit b9f9ff2 into main Sep 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant