Skip to content

feat: Add provider compatibility and recording evidence - #96

Draft
jiahuei wants to merge 8 commits into
mainfrom
feat/provider-compatibility-evidence
Draft

jiahuei wants to merge 8 commits into
mainfrom
feat/provider-compatibility-evidence

Conversation

@jiahuei

@jiahuei jiahuei commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Added explicit provider configuration for Together AI, Gemini, OpenRouter, Fireworks, and Bedrock, with protocol-specific destinations, credential isolation, and no silent endpoint switching.
  • Added strict reasoning/history extensions, native Google attachment and OpenRouter format preservation, unambiguous Gemini tool-index repair, balanced tool-stream lifecycles, and Together semantic-error recognition. Fireworks native continuation preserves observed Chat null placeholders and explicit nullable Messages content; Responses function tools may omit strict. Unsupported translation and invalid history fail explicitly.
  • Repaired recorder error-byte fidelity, partial-stream evidence, sanitization, and staging. Independent native SDK drivers cover all three protocols, including Messages bearer authentication, explicit provider controls, actual-output continuation, and completion validation. A reasoning-detail check stops when SDK accumulation corrupts repeated metadata. Fresh captures and source snapshots remain separate from unchanged historical matched fixtures from PR #76.
  • Added SDK parity, exact next-request bodies, translated semantic checks, corruption checks, credential/URL tests, and recorder/matcher regressions. Fireworks contributes 36 complete native conversations; OpenRouter contributes 13. Failed attempts remain visible, and original HTTP captures/manifests were not edited to make tests pass.

Related to #78. This remains a draft because the full issue is not complete. The support matrix distinguishes recorded, matched, synthetic, unsupported, and unknown coverage:

  • OpenRouter's public corpus contains 40 exchanges in 19 groups: 13 complete conversations and six unsuccessful attempts. nvidia/nemotron-3.5-lightning:free covers text and sequential tools across all three protocols and Chat parallel calls. All requests use explicit free IDs, zero price caps, disabled provider fallbacks, and no plugins; every reported cost was zero. Ten native histories replay fully; three non-streaming Chat histories reject provider reasoning:null. The 24 translated first-turn cases include explicit rejection of raw Responses reasoning under the summary profile. liquid/lfm-2.5-2.6b:free returned recorded HTTP 400 errors when reasoning was disabled. Two additional low-reasoning groups hit HTTP 429 and remain outside the repository because their responses contain an account identifier. The total live run used 43 requests.
  • Fireworks accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b completed text, sequential tools, and parallel tools across Chat, Responses, and Messages, each with both native modes and reasoning disabled/low. Its 47 groups contain 101 exchanges, including 11 incomplete attempts. All 36 completed conversations replay with exact native history; 72 translated first-turn cases preserve representable semantics or explicitly reject unsigned summaries.
  • Together Qwen/Qwen3.5-9B completed native SDK text and sequential tools in both modes, plus non-streaming parallel calls. Full Molto continuation is verified for non-streaming sequential/parallel tools. Recorded null reasoning/output-only histories reject; its reasoning_content output is unsupported by Together's translated reasoning dialect. Streaming parallel attempts returned only one call.
  • Gemini live requests returned HTTP 400 because this environment's location is unsupported. Its history/index behavior is covered offline. Bedrock live cells remain unknown because credentials/profile were unavailable.
  • Translated replay does not establish live translated requests or translated multi-turn continuation. Images and additional models remain gaps. Anthropic extra_content.anthropic continuation remains owned by feat: support hosted web search across Responses and Messages #74/fix: Preserve Anthropic continuation through extra_content #92.

Verification

  • uv run --extra test pytest: 2,195 passed on Python 3.12.9, offline/replay-only.
  • uv run pytest tests/core/test_openrouter_conversations.py tests/core/test_fireworks_conversations.py tests/core/test_recorded_provider_conversations.py -q --tb=short: 156 passed, including all 40 OpenRouter cases (native/rejection, translated, HTTP error, and SDK accumulation-failure parity).
  • Focused recorder suites: 22 native-driver and 29 Chat-driver tests passed.
  • uv run --extra lint ruff check --output-format github --config pyproject.toml .: passed.
  • uv run --extra lint ruff format --diff --config pyproject.toml .: passed.
  • npx prettier@3.8 --check .: passed. An initial formatting failure in authored catalog-selection metadata was corrected; original captures and manifests were unchanged.
  • git diff --check: passed. Original OpenRouter captures/manifests/source snapshots remained byte-identical to baseline 30f8b6d after replay.
  • Standalone native SDK driver runs retained exact settings and successful/incomplete outcomes in manifests. Public captures passed credential/account-identifier scans, decoded-byte inspection, source-hash checks, and byte comparisons against staging. No pytest recording mode was used. Bedrock live recording was not run because credentials/profile were unavailable.
  • Remote CI at 6002ba6: unit tests passed on Python 3.12, 3.13, and 3.14; both lint/formatting jobs passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant