Skip to content

test: Strengthen cassette parity checks and multi-turn replay - #95

Open
iAmir97 wants to merge 5 commits into
mainfrom
amer-improve
Open

iAmir97 wants to merge 5 commits into
mainfrom
amer-improve

Conversation

@iAmir97

@iAmir97 iAmir97 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Strengthened cassette comparisons independently of Molto's translators and collectors. Request and response checks now detect changed call IDs, opaque state, token limits, item ordering, incomplete reasons, and usage; stream checks validate item identity and lifecycle.
  • Matched conversations now send actual returned assistant output into subsequent public requests, preserving cumulative text, reasoning, tool calls, part boundaries, and encrypted snapshots. Negative tests verify that corrupted continuation history is rejected.
  • Expanded remote-image replay to all nine protocol pairs and added Responses and Messages callers using Chat upstream for parallel image tool results. Matching and request assertions keep tool results contiguous before relocated images and preserve image order. Added focused instruction and required-tool-choice projection checks.
  • Added provider-aware conversation discovery for interface, SDK parity, exception, and matcher tests. Discovery orders turns numerically, passes the selected provider explicitly, and rejects malformed or incompatible conversations without creating provider cross-products.
  • Corrected matched continuation fixtures, refreshed the incomplete Responses stream from an approved official recording, and added original Chat image-result recordings with provenance separate from their matched companions. Updated architecture, conventions, and cassette documentation.
  • Removed obsolete tests, 53 unused legacy cassettes, pytest recording infrastructure, unused utility/schema code, and unused dependencies. Migrated raw tool-argument preservation checks, retained incomplete-response and upstream-IR coverage, strengthened missing-credential checks, and made default async HTTP blocking unconditional. Translated clients now reuse one validated native request model.

Replay verifies the saved wire contract and does not establish additional live provider support. The raw-reasoning cache-write omission has SDK comparison coverage but lacks a faithful original establishing serving-provider provenance. Encrypted matched fixtures retain their documented single-issuer assumption.

Verification

  • .venv/bin/python -m pytest: passed, 1,515 tests in 20.05 seconds; replay only.
  • uv run --extra lint ruff check --output-format github --config pyproject.toml .: passed; existing inactive W3 selection warning only.
  • uv run --extra lint ruff format --diff --config pyproject.toml .: passed, 74 files already formatted.
  • npx prettier@3.8 --check .: passed.
  • git diff --check and git diff --cached --check: passed.

Signed-off-by: Amir Balwel <amoooori04@gmail.com>
Signed-off-by: Amir Balwel <amoooori04@gmail.com>
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Coverage

Coverage Report (Python 3.13) •
FileStmtsMissCoverMissing
__init__.py30100% 
types
   base.py25196%46
   normalized.py197199%240
upstream
   anthropic.py4252395%83, 89, 102–103, 219, 240, 333, 352, 374, 396, 454, 474, 518–519, 614, 623, 663, 684, 688, 714–715, 747, 749
   openai.py6253395%146, 148, 173, 181, 195, 210, 296, 325, 473, 481, 483, 486, 519–520, 572, 584, 600, 618, 668, 681, 755, 776, 935, 967, 1010–1011, 1134, 1176, 1180, 1193–1194, 1211, 1214
utils
   __init__.py00100% 
   io.py260100% 
   test.py790100% 
   types.py40100% 
TOTAL285716694% 

Tests Skipped Failures Errors Time
1629 0 💤 0 ❌ 0 🔥 30.008s ⏱️

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Coverage

Coverage Report (Python 3.14) •
FileStmtsMissCoverMissing
__init__.py30100% 
types
   base.py25196%46
   normalized.py161199%240
upstream
   anthropic.py4252395%83, 89, 102–103, 219, 240, 333, 352, 374, 396, 454, 474, 518–519, 614, 623, 663, 684, 688, 714–715, 747, 749
   openai.py6253395%146, 148, 173, 181, 195, 210, 296, 325, 473, 481, 483, 486, 519–520, 572, 584, 600, 618, 668, 681, 755, 776, 935, 967, 1010–1011, 1134, 1176, 1180, 1193–1194, 1211, 1214
utils
   __init__.py00100% 
   io.py260100% 
   test.py790100% 
   types.py40100% 
TOTAL281916694% 

Tests Skipped Failures Errors Time
1629 0 💤 0 ❌ 0 🔥 26.992s ⏱️

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Coverage

Coverage Report (Python 3.12) •
FileStmtsMissCoverMissing
__init__.py30100% 
types
   base.py25196%46
   normalized.py197199%240
upstream
   anthropic.py4252395%83, 89, 102–103, 219, 240, 333, 352, 374, 396, 454, 474, 518–519, 614, 623, 663, 684, 688, 714–715, 747, 749
   openai.py6253395%146, 148, 173, 181, 195, 210, 296, 325, 473, 481, 483, 486, 519–520, 572, 584, 600, 618, 668, 681, 755, 776, 935, 967, 1010–1011, 1134, 1176, 1180, 1193–1194, 1211, 1214
utils
   __init__.py00100% 
   io.py260100% 
   test.py790100% 
   types.py40100% 
TOTAL285716694% 

Tests Skipped Failures Errors Time
1629 0 💤 0 ❌ 0 🔥 53.900s ⏱️

Signed-off-by: Amir Balwel <amoooori04@gmail.com>
Signed-off-by: Amir Balwel <amoooori04@gmail.com>
Signed-off-by: Amir Balwel <amoooori04@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant