Publishing provider conformance: a machine-readable report, and how to collect one from providers we do not host

[aepfli]

Follow-up to #417, which asks in its open questions: "Who owns conformance claims? Is there a badge, a report format, a listing on openfeature.dev? Or is passing simply its own reward for now?" This issue is that question, split out so it can progress on its own timeline.

This is a design discussion, not a proposal to implement.

Status, 2026-09-11

The report format is no longer the substance of this issue. Following @toddbaert's comment below, #425 now defines only a small envelope and carries the results as Cucumber Messages rather than a format of our own. The sections below have been trimmed to what that leaves open: collection, trust and expiry. The minimized comments record how the format got there.

The opportunity

The provider conformance suite (Appendix F) has implementations in four languages, each of which already computes the data a public conformance report needs: the capability set a provider declares, and a per-scenario outcome. What is missing is a way to publish and collect reports. That would let openfeature.dev show, for any provider, what it actually supports — replacing "implements the provider contract" as an unverified claim with something a user can look up before choosing a provider.

The awkward part

Not every provider lives in an SDK-contrib repo. Many are in vendor organisations, on their own release cadence, and OpenFeature cannot run their suites — the backends and credentials are not ours. So any design has to work for providers we do not host and cannot execute.

Settled in #425

• Comparability. A report records the TCK implementation, its version and the spec revision its assets came from. The revision is a property of the TCK artifact: every implementation consumes the assets from this repository, via git submodule or, for Go, via the nested Go module, so the pin is always known. For the feature files the revision is corroborated by the Messages stream, which carries the executed source.
• Outcomes. Per-scenario outcomes come from Cucumber Messages. The envelope adds the declaration, which is an input to reading them, and a notApplicable set for capabilities a language cannot express — @numeric-coercion in JavaScript, which has no integer type. Collapsing that into "undeclared" would show every JS provider as missing something no JS provider can have.
• Known deviations. A provider can say "failed, known bug, tracked here", which is distinct from "chose not to". Three implementations had each invented a local shape for this.
• One provider, several reports. configuration keys them: flagd's RPC and in-process resolvers differ in whether they emit PROVIDER_STALE.

Never compute a conformance percentage

A single score rewards declaring fewer capabilities. Undeclare @stale and your percentage goes up. That is conformance theatre, and it exactly inverts the incentive the suite exists to create. A page should show a matrix of provider × capability with the distinct states visible, and no aggregate number. "This provider does not support configuration-change events" is genuinely useful to someone choosing a provider; hiding it behind a green 100% is actively harmful.

Open: trust

A JSON file in a vendor repository is a claim, not a fact. Three options:

Approach	Verdict
OpenFeature re-runs every suite centrally	does not scale — needs each vendor's backend and credentials
Naked self-report	worthless the moment anything is contested
Self-report with provenance	the only one that scales

Provenance meaning: the report is emitted only by CI, and carries the workflow run URL, commit SHA, and optionally a sigstore / GitHub OIDC attestation. The envelope has slots for all of these. Whatever we choose, the page should say plainly what it is showing: attested self-reports, not audits.

Open: discovery

Do not crawl. Suggested shape:

• a registry of pointers in an OpenFeature-owned repo — provider → report URL, added once by a vendor PR
• vendors self-publish the report to that URL (release asset, GitHub Pages, wherever)
• a scheduled job fetches every registered URL, validates against the schema, and rebuilds the page

The human-reviewed list stays small and stable; report content refreshes without anyone raising a PR. A /.well-known/openfeature-conformance.json convention could layer on later for vendors who prefer it.

Open: expiry

@dgenio's comment below proposes separating subject supersession (a newer provider release exists) from contract supersession (a newer TCK asset set exists), and keeping old reports visible as historical rather than deleting them. That needs the envelope to identify the tested subject strongly enough to make the comparison; provider.version is currently optional.

Suggested phasing

1. Schema in this repo. feat: add a schema for machine-readable provider conformance reports #425.
2. Emitters in each language TCK. Go, Python and JavaScript emit the earlier draft format and will be updated to the envelope once its shape is agreed. Java is in progress.
3. Registry + aggregator, designed once two or three real reports exist.
4. The page, last.

Questions

1. Should every TCK record the spec revision it vendored? Settled: the pin is always known. The envelope carries the commit and an optional release tag.
2. Is not-applicable the right fourth state? Settled as declaration.notApplicable in feat: add a schema for machine-readable provider conformance reports #425. Whether the specification should additionally scope capabilities per language, so a language-wide impossibility is recorded once rather than in every report, is still worth deciding.
3. Registry-of-pointers, or a well-known URL convention, or both?
4. How much provenance is proportionate — a CI run URL, or a full signed attestation?
5. Does a report expire? See the supersession model above.
6. Who owns the registry and the aggregator — this repo, openfeature.dev, or something new?
7. Should a report be published inside the provider's released artifact, and how strongly must the tested subject revision be identified?

[aepfli]

Phase 1 is now a PR: #425

The report schema is up as #425, stacked on #423. It was previously sitting inside #423 itself, which was wrong — the report format is a separate decision from the appendix and deserves its own review.

It covers only the part of this issue that seemed settled: what a report must contain for two reports to be comparable. Collection and trust stay open here.

The decisions most worth arguing with are called out in the PR description: four outcomes rather than two, per-scenario results required and required to be complete, and deliberately no aggregate score.

One update from running the suites, relevant to knownDeviation

The case for a distinct known-deviation concept has got stronger since this issue was written. It is now three implementations that each invented their own local shape for it:

• flagd narrows a float to an integer, in both Java and Go, in both resolvers of each — @strict-numeric-typing withheld in both adoptions.
• Python uses xfail(strict=True) for A boolean flag satisfies an Integer request: bool is a subclass of int python-sdk#619, so the deviation stays visible and un-hides itself when the SDK is fixed.
• JavaScript's multi-provider (the one in @openfeature/server-sdk, not the deprecated contrib package) replaces every child error code with GENERAL — 15 × TYPE_MISMATCH and 1 × FLAG_NOT_FOUND out of 29 scenarios. Everything else survives delegation intact.

That last one is worth noting for a reason beyond the count: it is not capability-gated and cannot be. Reporting the right error code is not an optional capability, so there is no tag to withhold, and the only options available today are "exclude the suite" or "ship a red build". Both lose the information. A report format that can say failed, known deviation, tracked here is the thing that would let it be published honestly instead of hidden.

Phase 2, emitters

Not started, deliberately. Writing four emitters against a schema still under discussion means rewriting four emitters. They follow agreement on the shape rather than preceding it.

[aepfli]

Phase 2 has started: the first emitter is up

go-sdk-contrib#944 emits this format from the Go TCK. Its reports validate against the schema in #425 with a Draft 2020-12 validator, which is the first evidence the schema is implementable rather than merely plausible.

Go was taken first deliberately, because it is the implementation the format most has to justify itself against. Concretely:

29 scenarios (29 passed)      <- godog's summary

Five of those twenty-nine did not run — they were skipped for undeclared capabilities. The emitted report for the same run says {"passed": 24, "not-declared": 5}, with a reason on each skip.

That is the argument for requiring per-scenario results, made concrete: the report does not fix the runner's summary, it makes the summary stop mattering. A consumer can check "a skipped scenario is never reported as passed" for themselves rather than trusting each runner to have applied it. pytest and jest-cucumber happen to report skips correctly, so without this the rule would hold in three languages and quietly fail in the fourth.

Two things the implementation settled

assetsTree is free in practice. The Go TCK already had a generator that copies the artifacts out of the submodule under a CI check; recording the revision and tree there was a few lines, and the existing check keeps them honest — a revision that disagrees with the artifacts beside it cannot be committed. This is the concrete reason the content digest was dropped in favour of the tree hash.

provider.name should be the provider's own metadata name, not the suite's. The suite name is chosen to read well in failure messages (flagd-rpc), which makes it the configuration. Worth stating in the schema documentation, because getting it backwards would silently produce reports that look provider-scoped and are not.

Still open here

Collection and trust — how OpenFeature receives reports from providers it does not host, and how far a self-published report should be believed — are untouched by either PR and remain the substance of this issue.

[aepfli]

Three emitters exist now, and they disagree about one thing

go-sdk-contrib#944, python-sdk-contrib#413 and js-sdk-contrib#1610. All three emit reports that validate against the schema in #425 under a Draft 2020-12 validator. Java is in progress.

Implementing the format turned out to be the real review of it. Three problems were clear enough to fix directly in #425 — a reason is now enforced for any outcome other than passed rather than merely expected; knownDeviation is available on a scenario as well as a capability; and capabilities now states that it is not a verdict. The fourth needs a decision here.

Scenario Outline rows are not portably identifiable

scenarios[] is keyed on (feature, name), and the schema assumes that pair is unique. It is not. Every row of a Scenario Outline shares one scenario name, and there is no field for the row.

This is not hypothetical. In the Python run, one row of "Requesting the wrong type returns the code default" has a different outcome from its ten siblings. Reported under a shared name, ten rows say passed and one says failed under the same key — a consumer cannot tell which case failed, and a naive one that builds a map keyed on (feature, name) silently keeps whichever row it saw last.

The three implementations have each solved it differently, because nothing told them not to:

	how an outline row is identified
Go	godog's own expansion of the scenario name
Python	the scenario name qualified with pytest's example id, e.g. … [boolean-flag-Integer-1]
JavaScript	jest-cucumber's expanded example title

All three are locally reasonable and none of them agree, so the same row of the same feature file appears under three different keys in three reports. Comparability across languages is the entire justification for having a shared format, and this is the one place it currently fails.

Worth noting that JavaScript hit a second, related problem: jest-cucumber never applies a scenario-name template to outline examples, so on the base branch four skipped outline rows were reported with no reason at all. Different bug, same root — outline rows are the case every runner treats as an afterthought, which is a good argument for the format being explicit about them rather than inheriting whatever each runner does.

Options

1. Add an optional example field to scenarioResult carrying the row's parameters as an object ({"flag": "boolean-flag", "type": "Integer"}), with the key becoming (feature, name, example). Precise and language-neutral, since the parameters come from the feature file rather than from the runner. Costs each implementation a little plumbing to get the example row out of its runner.
2. Add a required rowIndex — cheaper, but positional, so inserting a row into an Examples table silently re-keys every row after it.
3. Mandate a name-mangling convention, e.g. name [param1=value1, param2=value2]. No schema change, but it puts a string format in the normative text and every implementation has to reproduce it exactly, which is the kind of thing that drifts quietly.

I lean towards option 1: the parameters are the identity, and putting them in a field rather than smuggling them into a string is what lets a consumer group, diff and display them. It also makes the "one row of eleven failed" case legible, which is the case that exposed the problem.

Happy to be argued out of it — option 3 is meaningfully less work for the implementations, and if the view is that reports are read by humans more than by tooling, that changes the calculus.

[aepfli]

Decided: a Scenario Outline row is identified by its parameters

scenarios[] entries were keyed on feature + name, and every row of an outline shares one name. The type-mismatch matrix is eleven rows, so a report carried eleven entries separated by nothing but a duration measurement — and the Python implementation produced exactly the case that makes this matter, with one of the eleven failing and ten passing under one key.

An entry now carries the Examples row it came from, keyed by column header:

{
  "feature": "errors",
  "name": "Requesting the wrong type returns the code default",
  "example": { "key": "boolean-flag", "requested": "Integer", "default": "1" },
  "outcome": "failed",
  "reason": "error-code was 'GENERAL', expected 'TYPE_MISMATCH'"
}

Identity is (feature, name, example). Pushed to #425.

Why a field rather than a naming convention

The alternative was mandating a mangled name — name [key=boolean-flag, requested=Integer, default=1] — which needs no schema change and less work per implementation. It was rejected because it puts a separator, an ordering, and an escaping rule for values containing the separator into normative prose that every implementation has to reproduce byte-for-byte, and drift there stays invisible until two reports quietly fail to line up.

That is not a hypothetical. The three implementations had already diverged on this point before the field existed, each locally reasonable and none compatible with the others:

	what landed in name for the same row
Go	the bare scenario name, for all eleven rows
Python	the name plus pytest's example id
JavaScript	jest-cucumber's expanded example title

Values are the cell contents verbatim as strings, because Gherkin has no types — coercing "1" to a number would make the report disagree with the table it came from.

Related, and worth noting for anyone writing an emitter

JavaScript hit a second outline problem: jest-cucumber never applies a scenario-name template to outline rows, so four skipped rows were reported with no reason at all — a direct violation of Appendix F's "skipped, with the reason". Different bug, same root cause: outline rows are the case every runner treats as an afterthought. That is a good argument for the format being explicit about them rather than inheriting whatever each runner happens to do.

All four implementations are being updated to populate the field.

[dgenio]

I think the open question about whether a report "expires" becomes easier if we separate historical validity from current applicability.

A conformance report is evidence about a particular tested subject against a particular contract:

tested provider revision
× provider configuration
× SDK/runtime context where relevant
× TCK assets/spec revision
→ observed conformance result

Time passing does not by itself make that evidence false.

For example:

provider v1.4 / rpc
tested against TCK assets A
→ passed

If provider v1.5 is later released, the old report is still a valid statement about v1.4. It just is not evidence that v1.5 conforms.

Likewise, if the TCK moves from assets A to assets B and adds or changes a mandatory scenario, the report against A remains historically valid but does not establish conformance to B.

So I would avoid an expiresAt-style model and instead let the aggregator distinguish two independent forms of supersession:

1. subject supersession — the provider implementation/release being discussed is no longer the one that was tested;
2. contract supersession — the accepted/current TCK asset set is newer than the one exercised by the report.

That also suggests one schema concern in #425: the report should always identify the tested subject strongly enough to make this comparison. provider.version is currently optional, while provenance.commit answers a different question: where the report came from. In some setups that commit may identify the provider source too, but that is not guaranteed.

I would therefore keep these concepts separate:

provider identity       = which provider/configuration
tested subject revision = which exact release/artifact/source revision was exercised
report provenance       = who/what produced the report and where
TCK identity            = which contract/questions were exercised

I am not attached to requiring SemVer specifically. A provider version, source revision, artifact digest, or another stable provider-defined revision could all be reasonable depending on how the provider is distributed. The important invariant is that a report cannot silently float forward to a changed implementation.

For presentation, I would keep old reports rather than deleting them:

v1.4 / TCK A — passed (historical)
v1.5 / TCK B — no report / unverified

rather than allowing the v1.4 result to appear to describe v1.5.

This seems consistent with the rest of the design here: reports are claims with explicit provenance, not audits; configurations are not interchangeable; and absence/unsupported states should remain visible rather than being converted into reassuring aggregate numbers.

[toddbaert]

I'd really love not to have to invent our own format for this. It's another thing to maintain and version, and there are standards out there already that cover most of what we need.

Two worth considering before we commit to a bespoke schema:

• Cucumber Messages (https://github.com/cucumber/messages) (the ndjson protocol (https://github.com/cucumber/messages#ndjson)) seems like it could work; it's maintained, cross-language, JSON-schema'd, and preserves per-scenario outcome, the tags (i.e. our capability), and a skip message. cucumber-jvm and godog emit it natively; we'd have to implement emitters for jest-cucumber and pytest-bdd, but that's implementing an existing spec rather than inventing one, and it would probably benefit others

• JUnit/xUnit XML (common reference (https://github.com/testmoapp/junitxml)) is the universal format for this; every one of our four runners already emits it today. It's lossier (no tags, skip cause is free text), but we can carry the extra semantics with a conventions, I think.

The key idea, either way: "not-declared" and "not-applicable" are just sub-categories of "skipped". So we don't need a format with four statuses; we just need a standard skip plus a way to communicate the reason, and I think both formats can carry that (Messages carries it with the tags, which is why I think that might be better, but more impl work).

That leaves the only genuinely OpenFeature-specific piece as the capability summary, which is small and can be derived from the per-scenario results, instead of than being a concept we build... WDYT; is there a reason to prefer a custom format I'm missing?

[aepfli]

I fully agree and it makes sense. I will take a deeper look, this was for me just the next step, and i thought i add a feasibility to all my PRs for the TCK. I will explore the options, hopefully one of those are compatible with our whole stack or at least we can contribute upstream to make it work

[aepfli]

Sorry for the spam, i was not aware that my agent documented everything in the issues, I adapted my basic prompt and it should be better now.