Repository navigation
Conversation
Contributor
Coverage ReportProject coverage is 61.54%. Comparing base ( @@ Coverage Diff @@
## base head +/- ##
Coverage 61.54% 61.54% 0.00%
========================================
Files 37 37 0
Lines 3856 3856 0
========================================
Hits 2373 2373 0
========================================Generated at 2026-10-26T08:19:16.147Z by coverage |
xseman
force-pushed
the
test/e2e
branch
3 times, most recently
from
October 25, 2026 20:42
a5e40d2 to
97f8954
Compare
`make e2e` checks what a generated client puts on the wire and how it decodes the answer, for go-fetch, typescript-fetch and dart-fetch alike. The cases are language-neutral scenarios (testdata/e2e/scenarios) in the spec's own names and wire values: - a mock layer: a server built on kin-openapi validates each request and each scenario response against the feature spec and records the request, so a case checks the exact path, query, headers, cookies and body; - a backend layer: a strict FastAPI app (pydantic, uv-locked) whose served OpenAPI 3.1 spec is snapshotted, called through a client generated from it, echoing what it parsed. A runner per generator turns the scenarios into calls of its client the way its users write them (Go by reflection, TypeScript from the compiled sources, Dart with mirrors) and prints one JSON line per case. A gap of a generator is a skip with its reason on the case; a skip that starts passing fails. go-fetch passes every case; typescript-fetch skips 29 and dart-fetch 103, mostly clients that do not compile. The cookie array of parameters-object-styles is explode: false: the OpenAPI serialization table defines no exploded form for a cookie.
Six rounds of looking for realistic FastAPI and pydantic shapes a generated client gets wrong on the wire, each added to the backend with calls that show it (61 -> 81 cases): - discriminated unions as the body, a nullable field, map values and list items, behind a camelCase key, and a callable discriminator; - naive and aware date-times, times with offsets, negative durations, dates in lists, maps and query arrays; runners now run in a non-UTC zone (TZ=Europe/Bratislava); - required nullables and null against a non-null default, with the keys the client sent echoed back; - a typed form model; multipart labels echo joined by "|" so one part "x,y" no longer passes for two; - parameters whose names collide once camelCased, under a router prefix and a shared dependency, and ones named like a client's locals; - enums and Literals of mixed JSON types and with null. The gaps they found are skips with their reasons: go-fetch cannot hold a naive date-time or an enum without a type; typescript-fetch sends a date query array as Date.toString(), a naive date-time in UTC, an explicit null of an optional nullable as absent, and does not compile a union form field; the parser merges parameters of one camelCase name.
The runners number a repeated identifier as the generators now do (itemId, itemId2), so the FastAPI name-clash cases run. The naive date-time, mixed enum and null enum cases pass for go-fetch, and the explicit null of an optional nullable enum for typescript-fetch.
The backend streams from yield endpoints, which FastAPI describes with 3.2's itemSchema: JSON Lines of a model and of integers, events of a model, of a discriminated union over POST and of raw data with names and ids. The Go runner drains a streamed result into the list of its items.
…actly Besides the written scenarios, TestMock now makes cases from every JSON response and request body schema of the feature specs, the FastAPI spec (on the mock) and any spec in E2E_SPECS: all properties, only the required ones, nullables null, each union branch, edge scalars and extra keys, each value checked against its schema with kin-openapi. They go through the same mock and runners, and match gains an exact mode for them: no other keys, null is not absent. go-fetch passes all of them. A failing case of typescript-fetch or dart-fetch counts as failed but skips (valueLenient) until their gaps are fixed.
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
make e2e, an end-to-end test that runs generated clients against a server. It checks what each client sends on the wire and what it decodes from the answer. The same language-neutral cases run for go-fetch, typescript-fetch and dart-fetch.How it works
testdata/e2e/scenarios/*.yaml) are written in wire terms. Each one lists the operation and its args, what the server must receive, the answer, and the client's expected result. A known gap is askipwith its reason, and a skip that starts passing fails the test.openapi3filtervalidates every request and every scenario response, and records what arrived.testdata/e2e/backends/fastapi,uv.lock). Its served 3.1 spec is a snapshot thatmake e2e-updaterewrites. Endpoints echo what they parsed.internal/e2e/values_test.go): cases built from every JSON body schema of the feature specs and FastAPI, with all properties, only the required ones, nulls, each union branch, edge scalars and extra keys. They go through the same mock and runners and are compared exactly (no extra keys, null is not absent).E2E_SPECS=a,badds specs from outside the repo, such as GitHub, Kubernetes or Stripe.testdata/e2e/README.md):tscand Node;dart:mirrors.Local only for now (
E2E=1;go test ./...skips it). A warm run takes about 11 s. It needs uv, tsc with Node 22 or later, and Dart 3.13 or later (or setE2E_DART).FastAPI backend
Nine adversarial rounds grew the backend to 94 cases. It covers:
The generator bugs these rounds and the generated values found are fixed in #12.
Results
go-fetch is strict on both. With
E2E_SPECSit also passes 11,655 generated cases on GitHub, 2,239 on Kubernetes and 2,668 on Stripe (about 2 minutes). typescript-fetch and dart-fetch failures on generated values are counted but skipped until their fixes land.The skips record the gaps that remain:
go-fetch (2): integers past int64, and list headers sent comma-joined as the spec says, where FastAPI reads repeated lines.
typescript-fetch:
nullof an optional field;anyresult;They will be fixed in a follow-up PR.
dart-fetch: 18 clients do not compile (empty named parameter lists,
Type??, missing imports).parameters-object-styles.yaml: the cookie array is nowexplode: false, because OpenAPI defines no exploded form for a cookie.Checks
make test,make lint(0 issues) andmake e2e(dart-fetch on Dart 3.13.7) pass.