diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md index 25c9edd4..5077b8be 100644 --- a/specification/appendix-f-provider-conformance.md +++ b/specification/appendix-f-provider-conformance.md @@ -643,6 +643,61 @@ provider on a 32-bit accessor leaves undeclared. Nothing above 2^53 − 1 is ask cannot represent it, and what a provider owes a value that does not fit the requested accessor is the open question in [open-feature/spec#430](https://github.com/open-feature/spec/issues/430). +## The conformance report + +A run may emit a **conformance report**: a machine-readable document saying what was tested, what the +provider claimed, and where the results are. Its shape is fixed by +[`assets/provider-tck/report/conformance-report.schema.json`](./assets/provider-tck/report/conformance-report.schema.json), +and it is deliberately a transcription of that schema in every language rather than whatever each +language finds convenient, because the point of the format is that four implementations emit the +same thing. + +Emitting one is optional. A developer running the suite locally wants a pass or a failure, not a +document; CI publishing a claim wants the document. Nothing about a provider's conformance depends +on whether a report was written. + +The report is an **envelope**. It identifies the run and points at the results; it does not contain +them. The results are Cucumber Messages written alongside it, because per-scenario outcomes, tags, +Scenario Outline row identity and the executed feature source are all already specified there. +Restating them in a second format would create two places for the same fact to disagree. + +Two fields carry the interpretation the results cannot: + +- **`declaration.declared`** is every capability the configuration claims, as tags. It is an input to + reading the results rather than a summary of them: a skipped scenario says only that the question + was not put to this provider, and it is the declaration that says whether that is because the + provider declines the capability. Given the declaration and a scenario's tags, the reason for a + skip follows without being transported per scenario. +- **`knownDeviations`** are the gaps the provider author acknowledges. It sits beside the declaration + rather than inside it because it is not a claim about capabilities: an entry may concern a declared + capability, or a mandatory scenario belonging to no capability at all. What it qualifies is which + of the declaration's absences were decisions and which were defects — a distinction the results + cannot carry, because a skip looks the same either way. Omitted when empty, which is silence rather + than a claim of having no known gaps. + +### Provenance + +`provenance` is optional and records where a report came from: `runUrl`, `commit`, `timestamp` and an +optional signed `attestation`. + +It exists because **a report is a claim, not an audit**. Everything else in the document is what the +provider says about itself, and a consumer comparing two providers' reports — or deciding whether to +publish a badge from one — has no way to tell a report produced by a CI run against a tagged commit +from one produced by hand on a laptop. `provenance` is what lets that consumer decide how much to +trust the rest. + +It is optional rather than required because the facts in it are properties of the environment that +ran the suite, not of the provider or the TCK. A TCK implementation cannot invent them: outside CI +there is no run URL, and a working tree with uncommitted changes has no commit that describes what +was actually executed. A report that filled the field in anyway would be worse than one that omits +it, because the whole value of the field is that its contents can be checked. + +**No TCK implementation is required to emit it, and at the time of writing none does.** It is +specified here so that the first one to need it does not invent a second shape for the same facts, +and so that a reader who finds the field absent knows that means "not stated" rather than "this +implementation calls it something else". Where a suite runs in CI and the information is available, +populating it is encouraged; a consumer must treat its absence as unremarkable. + ## Implementing the suite in a language A TCK implementation is the language-specific harness around these three artifacts. Each obligation diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md index 2ce9b246..b8347f5f 100644 --- a/specification/assets/provider-tck/README.md +++ b/specification/assets/provider-tck/README.md @@ -16,11 +16,14 @@ These validate a **provider** against a real backend. For assets that validate a | [`gherkin/reason.feature`](./gherkin/reason.feature) | the standard resolution reasons, gated behind `@standard-reasons` | | [`flags/canonical-flags.json`](./flags/canonical-flags.json) | the flag set every scenario assumes | | [`openapi/control-api.yaml`](./openapi/control-api.yaml) | the HTTP surface a backend under test must expose | +| [`report/conformance-report.schema.json`](./report/conformance-report.schema.json) | the shape of a machine-readable conformance report | -## These three travel together +## The three inputs travel together A feature file that evaluates `boolean-flag` is meaningless without the flag definition, and a disconnect scenario is meaningless without the control endpoint that produces the disconnect. Changing one without the others breaks the suite in every language at once. +The report schema is not one of the three: it describes what a run *emits*, not what a run needs, and a suite that never writes a report is no less conformant for it. + ## Five properties that are load-bearing - **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. diff --git a/specification/assets/provider-tck/report/conformance-report.schema.json b/specification/assets/provider-tck/report/conformance-report.schema.json new file mode 100644 index 00000000..b1a1f6d5 --- /dev/null +++ b/specification/assets/provider-tck/report/conformance-report.schema.json @@ -0,0 +1,241 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://openfeature.dev/schema/provider-tck/conformance-report-1.json", + "title": "OpenFeature provider conformance report", + "description": "The machine-readable result of one run of the provider conformance suite (Appendix F) against one provider in one configuration. This document is an envelope: it identifies what was tested, what the provider claims, and where the executed results are. The results themselves are a standard format (Cucumber Messages), not something this specification defines. See https://github.com/open-feature/spec/issues/424.", + "type": "object", + "additionalProperties": false, + "required": [ + "schemaVersion", + "provider", + "sdk", + "tck", + "backend", + "declaration", + "results" + ], + "properties": { + "schemaVersion": { + "description": "The version of this schema the report conforms to. Integer-as-string so that a consumer can reject a report it does not understand rather than guessing.", + "type": "string", + "pattern": "^[1-9][0-9]*$" + }, + "provider": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "language" + ], + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "version": { + "type": "string" + }, + "language": { + "enum": [ + "go", + "java", + "javascript", + "dotnet", + "python", + "php", + "ruby", + "kotlin", + "swift", + "rust", + "other" + ] + }, + "configuration": { + "description": "Which configuration of the provider was tested, when a provider has more than one materially different mode. One provider may therefore produce several reports, and they are not interchangeable: flagd's RPC and in-process resolvers differ in whether they emit PROVIDER_STALE, so a report keyed on provider name alone would have to pick one and misrepresent the other.", + "type": "string" + }, + "repository": { + "type": "string", + "format": "uri" + } + } + }, + "sdk": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "version" + ], + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "version": { + "type": "string", + "minLength": 1 + } + } + }, + "tck": { + "description": "What asked the questions, and which questions. The spec revision is a property of the TCK artifact rather than an independent assertion: a given TCK build consumes exactly one revision of the assets.", + "type": "object", + "additionalProperties": false, + "required": [ + "implementation", + "version", + "specRevision" + ], + "properties": { + "implementation": { + "description": "Which TCK implementation produced this, e.g. 'go-sdk-contrib/tools/provider-tck'.", + "type": "string", + "minLength": 1 + }, + "version": { + "type": "string", + "minLength": 1 + }, + "specRevision": { + "description": "The open-feature/spec commit the executed assets came from.\n\nFor the feature files this is corroborated rather than merely asserted, because a Cucumber Messages stream carries the executed source and a consumer can compare it. For the canonical flag set and the control API description it stands alone: the stream says nothing about either, and a scenario asserting that integer-flag resolves to 10 depends entirely on the flag set it was run against. An earlier draft carried a hash of the whole asset directory to close that gap; it was dropped because every TCK implementation either reads the assets from the submodule directly or regenerates and diffs its vendored copy in CI, so drift is caught where it happens rather than re-detected in every report.", + "type": "string", + "minLength": 7 + }, + "specRelease": { + "description": "A human-readable spec release tag, when one applies. A commit is precise; a tag is what a person can reason about.", + "type": "string" + } + } + }, + "backend": { + "description": "What the provider was pointed at, and how the suite drove it. Present for every run: a provider with no backend still had its flag state manipulated somehow, and which of the two ways that was is the single most important thing a reader needs in order to know what the results are worth.", + "type": "object", + "additionalProperties": false, + "required": [ + "controlApi" + ], + "properties": { + "description": { + "description": "A short description of what was being controlled, as the control itself reports it. Free text, for a person: a container image and tag, an in-memory store, a named environment.", + "type": "string" + }, + "controlApi": { + "description": "How the backend was driven. 'http' means the normative control API, the contract that makes a conformance claim portable between languages; 'in-process' is the narrow allowance for providers with no backend, where the flag state is a data structure in the test process. A report claiming 'in-process' for a provider that does have a backend should be treated with suspicion -- the path it exercised is not the path the contract describes. Required, and required to come from the control rather than be inferred: nothing outside the control can tell which of the two it was, so a harness that guesses will eventually guess wrong about a custom control, and an absent value is not neutral -- every run is one or the other, so silence here reads as a claim that cannot be checked.", + "enum": [ + "http", + "in-process" + ] + } + } + }, + "declaration": { + "description": "The capability set this provider claims, as Gherkin tags including the leading at-sign. This is an INPUT to reading the results, not a summary of them, which is why it cannot be derived from the results payload and has to be stated here. A skipped scenario in the payload says the question was not put to this provider; the declaration is what says why. Given the declaration and a scenario's tags, most skips follow without needing a per-scenario explanation. NOT ALL OF THEM, and the exception is worth reading before building on this field: a capability may be absent because the provider declined it, or because the language's SDK cannot express it at all -- @numeric-coercion where the language has a single numeric type, @large-integers on a 32-bit accessor. Only the first says anything about the provider. The two are indistinguishable from the declaration alone, because an inexpressible capability is absent from every declaration in that language and no implementation permits declaring it. Appendix F records which capabilities that applies to per language, and the results payload carries the distinction per skip: a skip for an inexpressible capability says the SDK cannot ask the question, not that the provider declined. A consumer comparing providers across languages should read the skip reason rather than inferring from the declaration.", + "type": "object", + "additionalProperties": false, + "required": [ + "declared" + ], + "properties": { + "declared": { + "description": "Capabilities the provider declares AND that the executed suite gates on. A scenario tagged with anything absent from this list is expected to be skipped in the results payload.\n\nA capability that no executed scenario carries MUST NOT appear here, even when the provider genuinely supports it. Such a capability cannot produce a skip, so it plays no part in interpreting the results -- and listing it invites a reader to believe it was verified when nothing examined it. That is the vacuous conformance claim the capability vocabulary exists to prevent, and it is easy to reintroduce: an adopter declaring 'everything except X' picks up every reserved tag in the vocabulary along the way.\n\nAn implementation that finds a declared capability the suite does not test should say so where the adopter will see it -- a warning or a failed run -- rather than quietly passing it through to the report. Silence in the report and silence to the adopter are not the same thing.", + "type": "array", + "items": { + "type": "string", + "pattern": "^@[a-z0-9-]+$" + } + } + } + }, + "results": { + "description": "Where the executed results live, and in what format. The results themselves are deliberately NOT defined by this schema: per-scenario outcomes, tags, Scenario Outline row identity and the executed feature source are all already specified by Cucumber Messages, and defining them again here would create a second format to maintain and version and two places for the same fact to disagree.\n\nIt is referenced rather than inlined because a Messages stream carries the feature sources and so is far larger than this envelope, and because a consumer deciding whether it cares about a report should not have to download the whole run to find out.", + "type": "object", + "additionalProperties": false, + "required": [ + "format", + "location" + ], + "properties": { + "format": { + "description": "The results format. 'cucumber-messages' is the ndjson protocol at https://github.com/cucumber/messages -- the expected value, and the only one that carries the executed feature source and an exact Scenario Outline row identity. The others are admitted because a runner may not emit Messages yet, and are lossier: 'cucumber-json' identifies outline rows only by line number, and 'junit-xml' carries no tags at all, which leaves the capability that gated a skip unrecoverable.\n\nOne thing to know before reading a Messages stream: a testCaseFinished message carries no status. A scenario's outcome is the most severe testStepResult among its steps, hooks included, on the ordering UNKNOWN < PASSED < SKIPPED < PENDING < UNDEFINED < AMBIGUOUS < FAILED. That ordering is what makes a capability skip visible -- a gate that aborts in a before-hook produces a SKIPPED hook result outranking the PASSED steps it prevented -- and a consumer reading only testCaseFinished will see no outcome at all.", + "enum": [ + "cucumber-messages", + "cucumber-json", + "junit-xml" + ] + }, + "formatVersion": { + "description": "The version of the results format, for a format that has versions. Cucumber Messages does: implementations pin different releases -- one TCK builds against messages v21 while another's runner ships a much later one -- and the message types differ between them. Without this a consumer validating a stream has to guess which schema to validate against, and guessing wrong is worse than not checking, because a later schema accepts messages an earlier producer could not have emitted and an earlier one rejects messages that are perfectly valid. Record what the producer actually built against.", + "type": "string", + "minLength": 1 + }, + "location": { + "description": "Where to fetch the results: a URI, or a path relative to this document.", + "type": "string", + "minLength": 1 + }, + "digest": { + "description": "Digest over the results payload as 'sha256:', so a consumer can tell that what it fetched is what this envelope describes.", + "type": "string", + "pattern": "^sha256:[0-9a-f]{64}$" + } + } + }, + "knownDeviations": { + "description": "Deviations the provider acknowledges. Optional, and an empty list is not the same as an absent one: stating none is a claim, omitting the field is silence.", + "type": "array", + "items": { + "$ref": "#/$defs/knownDeviation" + } + }, + "provenance": { + "description": "Where this report came from. A report is a claim, not an audit, and this is what lets a consumer decide how much to trust it.", + "type": "object", + "additionalProperties": false, + "properties": { + "runUrl": { + "type": "string", + "format": "uri" + }, + "commit": { + "type": "string" + }, + "timestamp": { + "type": "string", + "format": "date-time" + }, + "attestation": { + "description": "An optional signed attestation (e.g. sigstore bundle) covering this document.", + "type": "string" + } + } + } + }, + "$defs": { + "knownDeviation": { + "description": "A gap the provider is known to have against something the specification does not treat as optional. It is distinct from an undeclared capability, which is a choice, and from a not-applicable one, which is impossible -- this is a defect that is acknowledged rather than hidden. Three of the four implementations needed it before it existed: flagd narrows floats to integers in two languages, and one Python scenario is marked as an expected failure against an SDK bug. Appendix F states which of the two legitimate shapes to prefer: declaring the capability and letting the scenario fail, over withholding the capability so its scenarios skip.", + "type": "object", + "additionalProperties": false, + "required": [ + "summary" + ], + "properties": { + "capability": { + "description": "The capability the deviation concerns, if it maps to one. Omitted when the deviation is against a mandatory scenario, which belongs to no capability.", + "type": "string", + "pattern": "^@[a-z0-9-]+$" + }, + "issue": { + "description": "Where the gap is tracked. Optional: an untracked deviation is still worth declaring, because naming the defect is what distinguishes it from a capability the provider chose to withhold. Prefer a tracked one as soon as there is somewhere to point at.", + "type": "string", + "format": "uri" + }, + "summary": { + "description": "What the gap is, for someone comparing providers. Required, because a deviation with no summary records that something is wrong without saying what, leaving a reader worse off than the bare skip or failure it accompanies.", + "type": "string", + "minLength": 1 + } + } + } + } +}