From ac48cd83c0f87e0e2845c3279fe306dbaddf95b7 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 13:30:18 +0200 Subject: [PATCH 01/46] feat(provider-tck): add the Python conformance suite for OpenFeature providers A conformance suite any Python provider can adopt to verify it implements the provider contract of the specification, and the Python implementation of the cross-language suite defined in Appendix F. It runs the same Gherkin, the same canonical flag set and the same control API as the Go and Java implementations. It uses pytest-bdd, the runner the flagd provider and the flagd testkit already use, so an adopting package gains no new test framework. Adoption is one fixture and one call. The step definitions ship as a pytest plugin registered through a pytest11 entry point, so there is no conftest.py to write and nothing to import for the vocabulary - pytest-bdd resolves steps through the fixture system, and fixtures from an installed plugin are visible everywhere. The feature files and flag set are packaged with the distribution, so adopting needs no git submodule. Capability gating uses pytest.skip from an autouse fixture, so a scenario whose capability was not declared is reported as skipped with the reason attached rather than silently passing. The gate keys off the node's markers rather than its requested fixtures: pytest-bdd resolves a step's fixtures lazily, so tck_config is not in request.fixturenames at setup time, and guarding on that silently disabled the gate. Two self-test suites, plus unit tests for what the Gherkin cannot assert about itself: the SDK's InMemoryProvider, and the TCK's own updatable one. The second exists because the first cannot exercise the configuration-change path at all. Findings, both confirmed by running the suite: * A boolean satisfies an Integer request. The client type-checks with isinstance(value, int) and bool subclasses int in Python, so boolean-flag requested as an Integer returns True with reason STATIC and no error code. This is Python-specific - the identical scenario passes in every other language - which is a fair argument for having more than one implementation. Tracked as open-feature/python-sdk#619, and marked xfail(strict=True) so it stays visible and un-hides itself once fixed. * InMemoryProvider cannot update its flag set, which Appendix A requires of an SDK in-memory provider. Only half the machinery is missing, since AbstractProvider already supplies emit_provider_configuration_changed, so ControllableInMemoryProvider is a small subclass rather than a reimplementation and should port back as a method. Tracked as open-feature/python-sdk#620. Verified locally: 56 passed, 7 skipped, 2 xfailed; ruff and mypy --strict clean. Part of https://github.com/open-feature/spec/issues/417 Signed-off-by: Simon Schrottner --- .release-please-manifest.json | 3 +- pyproject.toml | 2 + release-please-config.json | 9 + tools/openfeature-provider-tck/LICENSE | 201 ++++++++++ tools/openfeature-provider-tck/README.md | 199 ++++++++++ tools/openfeature-provider-tck/pyproject.toml | 69 ++++ .../contrib/tools/provider_tck/__init__.py | 129 ++++++ .../contrib/tools/provider_tck/capability.py | 91 +++++ .../contrib/tools/provider_tck/config.py | 169 ++++++++ .../tools/provider_tck/control-api.yaml | 368 ++++++++++++++++++ .../contrib/tools/provider_tck/control.py | 112 ++++++ .../provider_tck/features/errors.feature | 80 ++++ .../provider_tck/features/evaluation.feature | 59 +++ .../provider_tck/features/events.feature | 42 ++ .../provider_tck/features/lifecycle.feature | 33 ++ .../flag_data/canonical-flags.json | 82 ++++ .../contrib/tools/provider_tck/inprocess.py | 112 ++++++ .../contrib/tools/provider_tck/plugin.py | 104 +++++ .../contrib/tools/provider_tck/provider.py | 131 +++++++ .../contrib/tools/provider_tck/state.py | 146 +++++++ .../tools/provider_tck/steps/__init__.py | 11 + .../tools/provider_tck/steps/event_steps.py | 167 ++++++++ .../tools/provider_tck/steps/flag_steps.py | 238 +++++++++++ .../provider_tck/steps/provider_steps.py | 81 ++++ .../contrib/tools/provider_tck/values.py | 121 ++++++ .../tests/conftest.py | 37 ++ .../tests/test_controllable_conformance.py | 51 +++ .../tests/test_in_memory_conformance.py | 101 +++++ .../tests/test_in_process_control.py | 139 +++++++ 29 files changed, 3086 insertions(+), 1 deletion(-) create mode 100644 tools/openfeature-provider-tck/LICENSE create mode 100644 tools/openfeature-provider-tck/README.md create mode 100644 tools/openfeature-provider-tck/pyproject.toml create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control-api.yaml create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/errors.feature create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/evaluation.feature create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/events.feature create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/lifecycle.feature create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/flag_data/canonical-flags.json create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/__init__.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py create mode 100644 tools/openfeature-provider-tck/tests/conftest.py create mode 100644 tools/openfeature-provider-tck/tests/test_controllable_conformance.py create mode 100644 tools/openfeature-provider-tck/tests/test_in_memory_conformance.py create mode 100644 tools/openfeature-provider-tck/tests/test_in_process_control.py diff --git a/.release-please-manifest.json b/.release-please-manifest.json index 94d583941..6bb086204 100644 --- a/.release-please-manifest.json +++ b/.release-please-manifest.json @@ -8,5 +8,6 @@ "providers/openfeature-provider-unleash": "0.1.2", "tools/openfeature-flagd-api": "1.0.0", "tools/openfeature-flagd-core": "1.0.0", - "tools/openfeature-flagd-api-testkit": "0.1.0" + "tools/openfeature-flagd-api-testkit": "0.1.0", + "tools/openfeature-provider-tck": "0.1.0" } diff --git a/pyproject.toml b/pyproject.toml index 647571bc2..c1f1ce6bb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -16,6 +16,7 @@ dependencies = [ "openfeature-flagd-api", "openfeature-flagd-core", "openfeature-flagd-api-testkit", + "openfeature-provider-tck", ] [dependency-groups] @@ -43,6 +44,7 @@ openfeature-provider-unleash = { workspace = true } openfeature-flagd-api = { workspace = true } openfeature-flagd-core = { workspace = true } openfeature-flagd-api-testkit = { workspace = true } +openfeature-provider-tck = { workspace = true } [tool.uv.workspace] members = [ diff --git a/release-please-config.json b/release-please-config.json index a63354152..29cfce44b 100644 --- a/release-please-config.json +++ b/release-please-config.json @@ -99,6 +99,15 @@ "extra-files": [ "README.md" ] + }, + "tools/openfeature-provider-tck": { + "package-name": "openfeature-provider-tck", + "bump-minor-pre-major": true, + "bump-patch-for-minor-pre-major": true, + "versioning": "default", + "extra-files": [ + "README.md" + ] } }, "changelog-sections": [ diff --git a/tools/openfeature-provider-tck/LICENSE b/tools/openfeature-provider-tck/LICENSE new file mode 100644 index 000000000..261eeb9e9 --- /dev/null +++ b/tools/openfeature-provider-tck/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md new file mode 100644 index 000000000..d735a5c5e --- /dev/null +++ b/tools/openfeature-provider-tck/README.md @@ -0,0 +1,199 @@ +# OpenFeature Provider TCK (Python) + +A conformance suite any OpenFeature Python provider can adopt to verify that it implements the +provider contract of the specification. + +OpenFeature's central promise is that swapping providers does not change application behaviour. +Nothing verifies that today, and every provider tests differently — so "implements the provider +contract" is an unverified claim, and a behavioural difference between two providers is discovered +by the application that trips over it. + +This package is the Python implementation of [Appendix F][appendix-f]. It runs the same Gherkin +scenarios, against the same canonical flag set, driven through the same backend control API, as +every other language's TCK. That shared basis is the point: "conformant" only means something if the +question is identical everywhere. + +Tracking issue: [open-feature/spec#417][tracking]. + +## Status + +**Proof of concept.** The scenario set is a representative subset covering each architectural +mechanism once, not exhaustive coverage. Breaking changes should be expected. + +## Adopting it + +One fixture and one call. It uses **pytest-bdd**, the same runner the flagd provider and the flagd +testkit already use, so an adopting package gains no new test framework. + +```python +import pytest +from pytest_bdd import scenarios + +from openfeature.contrib.tools.provider_tck import ( + Capability, + TckConfig, + features_path, +) + + +@pytest.fixture(scope="session") +def tck_config(): + control = MyBackendControl() + return TckConfig( + name="my-provider", + control=control, + new_provider=lambda: MyProvider(control.address), + capabilities={Capability.EVENTS, Capability.OBJECT}, + ) + + +scenarios(features_path()) +``` + +There is **no `conftest.py` to write and nothing to import for the steps**. The step definitions +arrive through this package's pytest plugin, registered via a `pytest11` entry point, so installing +the package is all it takes. + +The TCK owns the whole lifecycle: registering the provider under a suite-scoped domain, awaiting +events, resetting the backend between scenarios, releasing it at the end. **If you find yourself +writing test infrastructure, that is a defect here rather than something for you to work around.** + +pytest-bdd generates one test per scenario — and one per row of a Scenario Outline — so failures +name a scenario and `-k` selects one as usual. The feature files and canonical flag set are packaged +with the distribution, so **you need no git submodule**. + +### Timings + +`TckConfig.event_timeout` is the knob that matters. Providers observe backend changes on wildly +different timescales — a streaming provider sees a configuration change in milliseconds, one that +polls every 30 seconds may need most of a poll interval. Set it to comfortably exceed your +worst-case detection latency, or the suite reports timeouts that are really just impatience. + +## Capabilities + +Not every provider implements every optional part of the contract. Each scenario exercising an +optional part carries a Gherkin tag, pytest-bdd turns that tag into a pytest marker, and a provider +declares what it supports. + +**A scenario whose capability was not declared is reported as skipped, with the reason — never as +passed.** A conformance suite that quietly goes green on scenarios it did not run is worse than no +suite at all, so `pytest.skip` carries the reason into the report: + +``` +SKIPPED provider does not declare capability @stale. + Declared: @events @object @strict-numeric-typing +``` + +| Capability | Tag | Meaning | +| --- | --- | --- | +| `Capability.EVENTS` | `@events` | emits lifecycle events at all | +| `Capability.STALE` | `@stale` | enters `STALE` and emits `PROVIDER_STALE` on backend loss | +| `Capability.CONFIGURATION_CHANGE` | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | +| `Capability.OBJECT` | `@object` | supports structured flag values | +| `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | +| `Capability.STRICT_NUMERIC_TYPING` | `@strict-numeric-typing` | does not coerce between integer and float | +| `Capability.TARGETING` | `@targeting` | reserved; no scenarios yet | +| `Capability.CACHING` | `@caching` | reserved; no scenarios yet | + +Untagged scenarios are mandatory and always run. `capabilities` defaults to everything — narrow it +rather than widening it: start from the default, run the suite, and remove only what your provider +genuinely cannot do. + +`@strict-numeric-typing` deserves a note, because unlike the others it is **not** an optional +feature. The specification requires `TYPE_MISMATCH` when the requested type cannot be satisfied, and +narrowing `0.5` to `0` loses information silently. It is a capability only so a provider with the +defect can adopt today and see the gap reported explicitly rather than being unable to adopt at all. +Not declaring it is an admission of a known bug. + +## Controlling the backend + +`BackendControl` is the single seam between the scenarios and whatever manipulates the backend. Step +definitions never talk to a backend directly, which is why the same Gherkin runs unchanged against a +containerised backend and against a provider manipulated in-process. + +**If your provider talks to a backend, drive it over the HTTP control API** — the document is +available as `control_api_spec()`. That API is the normative contract for those providers, and it is +what makes a conformance claim portable: another language's TCK drives the same endpoints against +the same stack and must get the same answers. + +Two of its requirements are easy to get wrong: + +- **Containers are never stopped or restarted mid-suite.** Unavailability is simulated *inside* the + running stack. Container orchestrators assign host ports dynamically and cannot reliably preserve + them across a restart, so restarting silently invalidates every provider already pointed at the + old port, and the failure looks like a flaky provider. +- **`/start` resets flag state; `/restart` preserves it.** An outage must be observable as a change + in availability, never as a change in flag values. + +### Providers with no backend + +An in-memory, environment-variable or file-based provider has nothing to connect to. Those may +control the backend in-process, where flag operations are direct manipulations of the provider's own +state. `InProcessControl` is the reference. + +This is a narrow allowance and the obvious thing to abuse. **A provider with an external backend +must use the control API.** Reaching into an external backend from inside the test process — a +test-only admin client, a shared database handle, a hook inside the provider — produces a suite that +passes while proving nothing, because the path it exercised is not the path the contract describes. + +Connection-dependent scenarios have no meaning without a connection, so a backend-less control +simply does not implement `ConnectionControl`, leaves `STALE` and `UNAVAILABLE_INIT` undeclared, and +those scenarios are skipped with their reason. + +## Findings + +Two, both confirmed by running the suite rather than by reading code. + +### 1. A boolean satisfies an Integer request + +`boolean-flag` evaluated through `get_integer_details` returns `True` with reason `STATIC` and **no +error code**, where the specification requires the code default and `TYPE_MISMATCH`. The client +type-checks with `isinstance(value, int)`, and `bool` is a subclass of `int` in Python. + +This is **Python-specific** — the identical scenario passes in every other language's suite, which +is a fair advertisement for having more than one implementation. Tracked as +[open-feature/python-sdk#619](https://github.com/open-feature/python-sdk/issues/619). + +The self-test marks that one row `xfail(strict=True)` with a pointer to the issue, so it stays +visible in the report and un-hides itself automatically once the SDK is fixed. + +### 2. The in-memory provider cannot update its flag set + +[Appendix A][appendix-a] requires an SDK's in-memory provider to support updating the flag set and +emitting `PROVIDER_CONFIGURATION_CHANGED`. Python's copies its mapping in the constructor and +exposes nothing to change it. Tracked as +[open-feature/python-sdk#620](https://github.com/open-feature/python-sdk/issues/620). + +Only half the machinery is missing — `AbstractProvider` already supplies +`emit_provider_configuration_changed` — which is why `ControllableInMemoryProvider` here is a small +subclass rather than a reimplementation, and why it should port back to the SDK as a method. + +## The self-tests + +| Suite | Subject | Why | +| --- | --- | --- | +| `test_in_memory_conformance` | the SDK's `InMemoryProvider` | reference adoption for a backend-less provider | +| `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | +| `test_in_process_control` | `InProcessControl` | pins what the Gherkin cannot assert about itself | + +``` +56 passed, 7 skipped, 2 xfailed +``` + +No Docker, no network, under a second. + +## Known gaps + +- **The assets are vendored, not submoduled.** `features/` and `flag_data/` are copies of + `specification/assets/provider-tck/` in [open-feature/spec][spec]. Changes belong there and are + copied here; a follow-up will source them from a submodule at build time, as + `openfeature-flagd-api-testkit` already does for the flagd test harness. +- **Evaluation context passthrough is unverifiable.** The scenarios build evaluation contexts but + cannot assert one *reached* the backend. That needs an echo operation on the control API. +- **No HTTP control client yet.** It arrives with the first containerised adopter. +- **Caching, hooks and flag metadata** are not covered. + +[appendix-a]: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md +[appendix-f]: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md +[spec]: https://github.com/open-feature/spec +[tracking]: https://github.com/open-feature/spec/issues/417 diff --git a/tools/openfeature-provider-tck/pyproject.toml b/tools/openfeature-provider-tck/pyproject.toml new file mode 100644 index 000000000..cdddc736c --- /dev/null +++ b/tools/openfeature-provider-tck/pyproject.toml @@ -0,0 +1,69 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "openfeature-provider-tck" +version = "0.1.0" +description = "OpenFeature provider conformance suite (TCK)" +readme = "README.md" +authors = [{ name = "OpenFeature", email = "openfeature-core@groups.io" }] +license = { file = "LICENSE" } +classifiers = [ + "License :: OSI Approved :: Apache Software License", + "Programming Language :: Python", + "Programming Language :: Python :: 3", + "Framework :: Pytest", +] +keywords = ["openfeature", "conformance", "tck", "feature-flags"] +dependencies = [ + "openfeature-sdk>=0.8.2", + "pytest>=8.4.0", + # Same runner the flagd provider and the flagd testkit already use, so an + # adopting module gains no new test framework. + "pytest-bdd>=8.1.0,<9.0.0", +] +requires-python = ">=3.10" + +[project.urls] +Homepage = "https://github.com/open-feature/python-sdk-contrib" + +# Shipping the step definitions as a pytest plugin is what keeps adoption to a +# single fixture: pytest-bdd resolves steps through the fixture system, and +# fixtures from an installed plugin are visible to every test, so an adopter +# never has to `from ... import *` to pull the vocabulary in. +[project.entry-points.pytest11] +openfeature_provider_tck = "openfeature.contrib.tools.provider_tck.plugin" + +[dependency-groups] +dev = [ + "coverage[toml]>=7.10.0,<8.0.0", + "mypy>=1.18.0,<2.0.0", + "poethepoet>=0.37.0", +] + +[tool.hatch.build.targets.wheel] +packages = ["src/openfeature"] + +[tool.mypy] +mypy_path = "src" +files = "src" +python_version = "3.10" +namespace_packages = true +explicit_package_bases = true +local_partial_types = true +allow_redefinition_new = true +fixed_format_cache = true +pretty = true +strict = true +disallow_any_generics = false + +[tool.coverage.run] +omit = ["tests/**"] + +[tool.poe.tasks] +test = "pytest tests" +test-cov = "coverage run -m pytest tests" +cov-report = "coverage xml" +cov = ["test-cov", "cov-report"] +mypy = "mypy" diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py new file mode 100644 index 000000000..31e9d39d3 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py @@ -0,0 +1,129 @@ +"""The OpenFeature Provider Conformance Suite (TCK) for Python. + +The suite answers one question: does this provider map its backend onto the +OpenFeature provider contract correctly? It is the Python implementation of +`Appendix F`_ of the specification, and it runs the same Gherkin scenarios, +against the same canonical flag set, that every other language's TCK runs. That +shared basis is the whole point -- "conformant" only means something if the +question is identical everywhere. + +**What a provider author writes.** One fixture and one call:: + + import pytest + from pytest_bdd import scenarios + + from openfeature.contrib.tools.provider_tck import ( + Capability, + InProcessControl, + TckConfig, + features_path, + ) + + @pytest.fixture(scope="session") + def tck_config(): + control = InProcessControl() + return TckConfig( + name="my-provider", + control=control, + new_provider=control.new_provider, + capabilities={Capability.EVENTS, Capability.OBJECT}, + ) + + scenarios(features_path()) + +``scenarios()`` is pytest-bdd's own, called directly rather than wrapped: it +injects the generated tests into the *calling module* by walking the stack, so a +convenience wrapper around it would deposit them inside this package instead. + +The step definitions arrive through this package's pytest plugin, so there is +nothing to import for them and no ``conftest.py`` to write. Everything else -- +registering the provider, awaiting events, resetting the backend between +scenarios, tearing down -- belongs to the TCK. If you find yourself writing test +infrastructure, that is a defect here rather than something for you to work +around. + +.. _Appendix F: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md +""" + +from __future__ import annotations + +import importlib.resources + +from .capability import ALL_CAPABILITIES, Capability +from .config import TckConfig +from .control import ( + BackendControl, + ConnectionControl, + UnsupportedControlError, +) +from .inprocess import InProcessControl +from .provider import ( + CHANGING_FLAG_KEY, + ControllableInMemoryProvider, + canonical_flag_set, +) + +__all__ = [ + "ALL_CAPABILITIES", + "CHANGING_FLAG_KEY", + "BackendControl", + "Capability", + "ConnectionControl", + "ControllableInMemoryProvider", + "InProcessControl", + "TckConfig", + "UnsupportedControlError", + "canonical_flag_set", + "canonical_flags_json", + "control_api_spec", + "features_path", +] + +# NOTE ON THE SOURCE OF TRUTH +# +# The files under features/ and flag_data/ are NOT owned by this repository. +# They are copies of the language-agnostic conformance artifacts defined in +# open-feature/spec under specification/assets/provider-tck/. They are vendored +# here so adopting this TCK never requires a git submodule of your own. Changes +# belong in open-feature/spec first and are copied here -- editing them locally +# forks the definition of conformance, which is the one thing this suite exists +# to prevent. See https://github.com/open-feature/spec/issues/417. + +_PACKAGE = "openfeature.contrib.tools.provider_tck" + + +def features_path() -> str: + """Return the directory holding the canonical feature files. + + Packaged with this distribution, so a consumer needs no submodule and no + particular directory layout. Hand it to pytest-bdd's ``scenarios()``, which + accepts an absolute path:: + + scenarios(features_path()) + + pytest-bdd generates one test per scenario -- and one per row of a Scenario + Outline -- so failures name a scenario and ``-k`` selects one as usual. + """ + return str(importlib.resources.files(_PACKAGE) / "features") + + +def canonical_flags_json() -> str: + """Return the canonical flag set as raw JSON, in the flagd flag-definition format. + + This is the flag set every scenario assumes, and a backend under test must + serve an equivalent one. The format is not what matters -- the keys, types, + variant names and resolved values are. Seed them however your backend seeds + flags. + + Exposed so an adopting provider can seed a backend from the canonical + definition rather than transcribing it, transcription being the usual way + the two drift apart. + """ + ref = importlib.resources.files(_PACKAGE) / "flag_data" / "canonical-flags.json" + return ref.read_text(encoding="utf-8") + + +def control_api_spec() -> str: + """Return the OpenAPI document a containerised backend under test must implement.""" + ref = importlib.resources.files(_PACKAGE) / "control-api.yaml" + return ref.read_text(encoding="utf-8") diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py new file mode 100644 index 000000000..044908648 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py @@ -0,0 +1,91 @@ +"""Optional parts of the provider contract, and the Gherkin tags that gate them.""" + +from __future__ import annotations + +from enum import Enum + +__all__ = ["Capability"] + + +class Capability(str, Enum): + """An optional part of the OpenFeature provider contract. + + Not every provider implements every part of the specification. A provider + backed by a static file has no meaningful notion of going stale; one with no + streaming transport cannot emit configuration-change events. Rather than + forcing such providers to fail scenarios they were never going to satisfy, + each declares what it supports through :attr:`TckConfig.capabilities`. + + Every capability corresponds to exactly one Gherkin tag. pytest-bdd turns + those tags into pytest markers, and a scenario carrying a marker whose + capability was not declared is skipped with the reason reported -- never + passed. A conformance suite that quietly goes green on scenarios it did not + run is worse than no suite at all. + + Scenarios with no capability tag are mandatory and always run. + """ + + EVENTS = "events" + """Provider emits lifecycle events at all, at minimum ``PROVIDER_READY``.""" + + STALE = "stale" + """Provider enters ``STALE`` and emits ``PROVIDER_STALE`` when it loses its backend.""" + + CONFIGURATION_CHANGE = "configuration-change" + """Provider detects configuration changes and emits ``PROVIDER_CONFIGURATION_CHANGED``.""" + + OBJECT = "object" + """Provider supports structured (object) flag values.""" + + UNAVAILABLE_INIT = "unavailable" + """Provider reports an error state promptly against a backend it cannot reach.""" + + STRICT_NUMERIC_TYPING = "strict-numeric-typing" + """Provider keeps the integer and float types distinct instead of coercing between them. + + Unlike every other entry here this is not an optional feature. The + specification requires a provider to report ``TYPE_MISMATCH`` when the + requested type cannot be satisfied, and narrowing ``0.5`` to ``0`` to satisfy + an integer request loses information silently -- the worst failure mode a + feature flag has, because the application sees a plausible value and no + error at all. + + It is a capability only so that a provider with this defect can adopt the + suite today and see the gap reported as an explicit skip, rather than being + unable to adopt at all. Not declaring it is an admission of a known bug, not + a design choice. Declare it as soon as the provider is fixed. + """ + + TARGETING = "targeting" + """Reserved. No scenario carries this tag: targeting is backend evaluation logic.""" + + CACHING = "caching" + """Reserved; no scenario carries this tag yet.""" + + @property + def tag(self) -> str: + """Return the Gherkin tag, with its leading at-sign, that gates this capability.""" + return f"@{self.value}" + + def __str__(self) -> str: + return self.tag + + +ALL_CAPABILITIES: frozenset[Capability] = frozenset(Capability) +"""Every capability the TCK recognises. + +A reasonable starting point for a new adoption: declare everything, run the +suite, and remove only what the provider genuinely cannot do. Narrowing from the +full set surfaces gaps; widening towards it hides them. +""" + +_BY_MARKER: dict[str, Capability] = {c.value: c for c in Capability} + + +def capability_for_marker(name: str) -> Capability | None: + """Map a pytest marker name onto the capability it gates, if any. + + A marker that does not name a capability gates nothing, which is what lets + the canonical feature files carry organisational tags freely. + """ + return _BY_MARKER.get(name) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py new file mode 100644 index 000000000..468a4921c --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py @@ -0,0 +1,169 @@ +"""The contract a provider author implements to run the suite.""" + +from __future__ import annotations + +from collections.abc import Callable, Iterable +from dataclasses import dataclass, field + +from openfeature.provider import FeatureProvider + +from .capability import ALL_CAPABILITIES, Capability +from .control import BackendControl + +__all__ = ["ProviderFactory", "TckConfig"] + +ProviderFactory = Callable[[], FeatureProvider] +"""Creates the provider under test. + +A factory rather than a single instance because each scenario gets its own +provider, and because a provider often cannot be configured before the suite +starts -- a container stack's host ports do not exist until it is up. +""" + +DEFAULT_EVENT_TIMEOUT = 12.0 +DEFAULT_READY_TIMEOUT = 30.0 + + +@dataclass(frozen=True) +class TckConfig: + """Everything the TCK needs to test one provider. + + An adopting module supplies this through a session-scoped ``tck_config`` + fixture; the TCK owns everything else -- registering the provider, awaiting + events, resetting the backend between scenarios, tearing down. If you find + yourself writing test infrastructure, that is a defect in this package + rather than something for you to work around. + """ + + name: str + """Identifies the suite in test output, and scopes the OpenFeature domain + the TCK registers providers under so two suites in the same session do not + observe each other's providers. + + Use something that reads well in a failure message: ``"flagd-rpc"``, + ``"in-memory"``. + """ + + control: BackendControl + """The seam through which the TCK manipulates the backend. + + See :class:`~.control.BackendControl` for which implementation is right for + your provider. The short version: a provider with a real backend drives it + over the HTTP control API; a provider with no backend at all may control it + in-process. + """ + + new_provider: ProviderFactory + """Creates the provider under test, against a backend that is already + running and seeded with the canonical flag set. Called once per scenario. + + Return a configured but uninitialised provider; the TCK initialises it. + """ + + new_unavailable_provider: ProviderFactory | None = None + """Creates a provider pointed at a backend that does not exist. + + Used by the initialisation-failure scenarios, which assert that a provider + unable to reach its backend settles into ``ERROR`` rather than hanging or + raising out of registration. + + Point it at a closed port on localhost. Do not point it at the backend under + test -- that must stay up, and simulated outages belong to :attr:`control`. + Configure a short connection deadline: the scenario allows a bounded time + for the error, and a provider with a 30-second connect timeout will not make + it. + + Required only if :attr:`capabilities` includes + :attr:`Capability.UNAVAILABLE_INIT`. Leaving both out is the honest + configuration for a provider with no backend, and those scenarios are then + skipped with the reason reported. + """ + + capabilities: frozenset[Capability] = field(default=ALL_CAPABILITIES) + """Which optional parts of the provider contract this provider supports. + + Scenarios tagged with an undeclared capability are reported as skipped with + the reason, never as passed. Defaults to everything; narrow it rather than + widening it. + """ + + event_timeout: float = DEFAULT_EVENT_TIMEOUT + """Seconds to wait for a provider event. + + The single most important knob for a provider author, because providers + observe backend changes on wildly different timescales. A streaming provider + sees a configuration change in milliseconds; one polling every 30 seconds + may need most of a poll interval. Set it to comfortably exceed your + worst-case detection latency, or the suite reports timeouts that are really + just impatience. + + Scenarios can tighten this with the explicit ``within {int}ms`` step, which + always wins over this value. + """ + + ready_timeout: float = DEFAULT_READY_TIMEOUT + """Seconds to wait for a provider to reach ``READY`` during initialisation.""" + + def __post_init__(self) -> None: + problems: list[str] = [] + + if not self.name: + problems.append( + "name is required: it scopes the OpenFeature domain and identifies " + "the suite in test output" + ) + if self.control is None: + problems.append( + "control is required: see BackendControl for which implementation " + "fits your provider" + ) + if self.new_provider is None: + problems.append("new_provider is required: the TCK has nothing to test without it") + + # Normalise whatever iterable the caller passed into a frozenset, so a + # set literal, a list or a generator all behave the same. + object.__setattr__(self, "capabilities", frozenset(self.capabilities)) + + unknown = [c for c in self.capabilities if not isinstance(c, Capability)] + if unknown: + problems.append( + f"unknown capabilities {unknown!r}: capabilities are the members of " + f"the Capability enum" + ) + + if Capability.UNAVAILABLE_INIT in self.capabilities and self.new_unavailable_provider is None: + problems.append( + "capabilities declares Capability.UNAVAILABLE_INIT but " + "new_unavailable_provider is None: the @unavailable scenarios need a " + "provider pointed at a backend that does not exist. Supply one, or " + "remove the capability so those scenarios are skipped with a reason" + ) + + if problems: + joined = "\n - ".join(problems) + msg = f"invalid TckConfig:\n - {joined}" + raise ValueError(msg) + + @property + def domain(self) -> str: + """The OpenFeature domain this suite registers its providers under. + + Suite-scoped rather than scenario-scoped on purpose. Registering a new + provider in the same domain replaces the previous one; a fresh domain + per scenario would leave every provider of the suite registered, which + for a provider holding a network connection means leaking one connection + per scenario. + """ + return f"provider-tck/{self.name}" + + def declares(self, capability: Capability) -> bool: + return capability in self.capabilities + + @property + def sorted_capabilities(self) -> list[str]: + return sorted(c.tag for c in self.capabilities) + + +def capabilities_of(values: Iterable[Capability]) -> frozenset[Capability]: + """Convenience for building a capability set from any iterable.""" + return frozenset(values) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control-api.yaml b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control-api.yaml new file mode 100644 index 000000000..fd9bc7000 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control-api.yaml @@ -0,0 +1,368 @@ +openapi: 3.0.3 + +info: + title: OpenFeature Provider TCK — Backend Control API + version: 0.0.1 + description: | + The control API that a **backend under test** must expose so the OpenFeature + Provider TCK can drive it. + + The TCK verifies the *provider contract*: how a provider maps backend + responses to typed resolution details, lifecycle states and events. To do + that it must be able to put the backend into specific states on demand — + running, unreachable, reconfigured. This document standardises how. + + This specification is derived from the control endpoints already implemented + by [`flagd-testbed`](https://github.com/open-feature/flagd-testbed)'s + "launchpad" server, which is the reference implementation. + + ## Where this document should live + + This file currently ships inside the Java `provider-tck` artifact, but it is + not a Java artifact: it is a language-agnostic contract that every language's + TCK must implement identically, and that backend vendors implement in + whatever language their testbed is written in (Go, for flagd). + + It therefore belongs in the OpenFeature **spec** repository + (`open-feature/spec`), alongside the canonical Gherkin feature files and the + canonical flag set. Those three artifacts are a single unit — a feature file + that evaluates `boolean-flag` is meaningless without the flag definition, and + a disconnect scenario is meaningless without the endpoint that produces the + disconnect. Splitting them across repositories would let them drift. + + Each language's TCK then vendors the spec repo (git submodule or equivalent) + and packages these files into its own distribution format, so that adopting a + TCK never requires a consumer to check out a submodule of their own. + + ## Conformance language + + The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT and MAY are to be + interpreted as described in RFC 2119. + + Each operation below is tagged **REQUIRED** or **OPTIONAL**. A backend that + implements every REQUIRED operation can run the full TCK. OPTIONAL operations + have a defined fallback that the TCK applies automatically, so omitting them + costs nothing but precision. + + --- + + ## Normative requirement 1 — the no-container-restart invariant + + > **Container lifecycle operations MUST NOT be used to simulate backend + > unavailability. Backend unavailability MUST be simulated from inside the + > running stack.** + + The TCK starts the vendor's Docker Compose stack **once per test suite** and + reads the dynamically mapped host ports. Testcontainers cannot reliably + preserve mapped ports across a container stop/start in all language + bindings — a restarted container generally comes back on a *different* host + port, which silently invalidates every provider instance already pointed at + the old one. Any TCK implementation in any language hits this, so the + constraint is part of the contract rather than a Java detail. + + Therefore an implementation of `/stop`, `/restart` or any other outage + simulation MUST achieve the outage by one of: + + * killing or suspending the backend **process** inside its container + (the reference behaviour — this is what flagd-testbed does); + * a proxy in the stack refusing or blackholing connections + (e.g. a toxiproxy toxic, an envoy `direct_response`); + * an in-container firewall or socket-level block. + + An implementation MUST NOT `docker stop`, `docker kill`, `docker rm` or + recreate any container in the stack while the suite is running. The stack is + brought up before the first scenario and torn down after the last one, and + the mapped ports MUST remain stable for that entire window. + + --- + + ## Normative requirement 2 — flag state semantics across outages + + Outage simulation and flag-state seeding are orthogonal, and the TCK relies + on that separation for scenario isolation: + + * `POST /start` **MUST** (re)seed flag state to the baseline defined by the + named configuration. Any mutation previously applied by `POST /change` + MUST be discarded. This is what makes `/start` usable as a reset. + * `POST /restart` and a `POST /stop` followed by a `POST /start` **of the + same configuration** MUST leave the backend serving the same baseline + flag state it served before the outage. An outage MUST NOT be observable + as a change in flag *values* — only as a change in *availability*. + * `POST /change` mutations persist until the next `/start` or `/reset`. + + --- + + ## Normative requirement 3 — compose stack conventions + + The backend under test is delivered as a **Docker Compose stack**, not a + single image, so vendors can compose proxies, edge services or several + containers. The TCK only relies on these conventions: + + * One service — by default named `backend`, overridable by the provider + author — exposes the control API on container-internal port `8080` + (also overridable). + * The same stack exposes whatever port(s) the provider connects to. + * **All external ports are dynamically mapped.** A stack MUST NOT pin host + ports; the TCK discovers them after startup and hands them to the + provider factory. + * The stack MAY contain any number of additional services. + + --- + + ## Known gap — evaluation context passthrough + + There is currently no operation for asserting that an evaluation context sent + by the provider actually reached the backend intact. Verifying that requires + an echo mechanism (e.g. `GET /last-evaluation` returning the most recent + request the backend received). Until such an operation exists, context + passthrough is out of scope for the TCK. + + license: + name: Apache 2.0 + url: https://www.apache.org/licenses/LICENSE-2.0 + +servers: + - url: http://{host}:{port} + description: | + Resolved at runtime from the Compose stack. `host` is the Docker host and + `port` is the dynamically mapped host port for the control service's + internal port 8080. + variables: + host: + default: localhost + port: + default: "8080" + +tags: + - name: lifecycle + description: Start and stop the backend process. + - name: availability + description: Simulate outages without touching containers. + - name: flags + description: Seed and mutate flag configuration. + - name: health + description: Readiness of the control API itself. + +paths: + + /start: + post: + tags: [lifecycle] + operationId: start + summary: "[REQUIRED] Start the backend and seed flags to a named baseline" + description: | + Starts the backend process using the named configuration and seeds flag + state to that configuration's baseline. + + MUST be idempotent in the sense that calling it while the backend is + already running is not an error: the implementation restarts the process + (or otherwise ensures it is running) with the requested configuration. + + Because this operation resets flag state, the TCK uses it as its default + scenario-isolation mechanism when `/reset` is not implemented. + + The set of valid configuration names is vendor-defined. Every + implementation MUST support the name `default`, which MUST serve the + canonical flag set the TCK's feature files assume. + + Reference implementation: flagd-testbed launches the `flagd` binary with + the config file of that name from `launchpad/configs` and rewrites + `/flags/allFlags.json`. + parameters: + - name: config + in: query + required: false + description: | + Name of the configuration to start with. Defaults to `default`. + schema: + type: string + default: default + example: default + responses: + "200": + description: Backend started and flag state seeded. + "400": + description: Unknown configuration name. + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + + /stop: + post: + tags: [availability] + operationId: stop + summary: "[REQUIRED] Make the backend unreachable" + description: | + Makes the backend unreachable to the provider, simulating an outage. + + **MUST NOT stop the container.** See normative requirement 1. The + reference implementation kills the flagd process while its container + keeps running. + + The backend stays unreachable until a subsequent `POST /start`. Calling + `/stop` when the backend is already stopped MUST succeed. + + The TCK uses this to drive providers into `STALE` and `ERROR` states and + to assert `PROVIDER_STALE` / `PROVIDER_ERROR` events. + responses: + "200": + description: Backend is now unreachable; container still running. + + /restart: + post: + tags: [availability] + operationId: restart + summary: "[REQUIRED] Simulate an outage of a bounded duration" + description: | + Makes the backend unreachable, waits `seconds`, then starts it again with + the configuration currently in effect. + + Flag state MUST be preserved across the outage — see normative + requirement 2. This is what distinguishes `/restart` from + `/stop` + `/start`: the former is an availability event, the latter is + also a reset. + + This operation MAY return as soon as the outage has begun rather than + blocking for the full duration; the TCK does not rely on the response + being delayed. It awaits provider events instead. + + The TCK uses this for the disconnect/reconnect scenarios: `STALE` → + `PROVIDER_STALE`, then back to `READY` → `PROVIDER_READY`. + parameters: + - name: seconds + in: query + required: false + description: | + How long the backend stays unreachable. Defaults to 5. + + Providers differ enormously in how fast they notice an outage — + a streaming provider may see it in milliseconds while a polling + provider needs up to a full poll interval. Feature files therefore + parameterise this value and provider authors tune the matching + await timeouts. + schema: + type: integer + format: int32 + minimum: 0 + default: 5 + example: 5 + responses: + "200": + description: Outage started (and, for blocking implementations, ended). + + /change: + post: + tags: [flags] + operationId: change + summary: "[REQUIRED] Mutate flag configuration so the provider observes a change" + description: | + Mutates the flag configuration such that a conforming provider observes a + configuration change and, on re-evaluation, resolves a **different value** + for the affected flag. + + The implementation MUST: + + * change the resolved value of the flag with key `changing-flag`; + * do so without restarting the backend process, so that a provider sees + a configuration-change signal rather than a reconnect; + * make the change durable until the next `/start` or `/reset`. + + The implementation SHOULD toggle between exactly two known values so that + repeated calls are meaningful and the test remains deterministic + regardless of how many times it has run against the same stack. The + reference implementation toggles `changing-flag`'s `defaultVariant` + between `foo` and `bar`. + + The TCK uses this to assert `PROVIDER_CONFIGURATION_CHANGED`, that the + changed flag key appears in the event payload, and that a subsequent + evaluation returns the new value. + responses: + "200": + description: Flag configuration mutated. + + /reset: + post: + tags: [flags] + operationId: reset + summary: "[OPTIONAL] Restore the seeded baseline without an outage" + description: | + Restores flag state to the baseline of the configuration currently in + effect, discarding any mutation applied by `/change`, **without** making + the backend unreachable at any point. + + This is the preferred scenario-isolation primitive: unlike `/start` it + causes no availability blip, so it cannot inject spurious lifecycle + events into the next scenario. + + **Scope.** This operation resets flag state only. It MUST NOT be + expected to start a backend that is currently stopped — that is what + `/start` is for. A TCK therefore uses `/reset` only when the backend is + known to be running, and `/start` otherwise. The reference client tracks + this: `/stop` and `/restart` mark the backend as possibly-unreachable, so + the scenario that follows either of them is prepared with `/start`. + + **Fallback when not implemented.** A backend that does not implement this + operation MUST respond `404` or `501`. The TCK then falls back to + `POST /start?config={defaultConfig}`, which resets flag state at the cost + of a process restart. The fallback is detected once per suite and cached. + + Implementing `/reset` is RECOMMENDED for providers whose reconnect + behaviour makes the `/start` blip hard to distinguish from a real event. + responses: + "200": + description: Flag state restored to the baseline. + "404": + description: Not implemented; the TCK falls back to `/start`. + "501": + description: Not implemented; the TCK falls back to `/start`. + + /healthz: + get: + tags: [health] + operationId: health + summary: "[OPTIONAL] Readiness of the control API" + description: | + Reports whether the control API is ready to accept commands. + + **Fallback when not implemented.** Readiness defaults to "the control + port accepts a TCP connection", which the TCK establishes with a + Testcontainers listening-port wait strategy before the first scenario. A + `404` here is therefore not a failure, and the reference implementation + does not serve this path. + + Note this reports the health of the **control API**, not of the backend. + The backend is deliberately unhealthy during outage scenarios while the + control API must stay reachable — otherwise the TCK could not end the + outage. + responses: + "200": + description: Control API ready. + content: + application/json: + schema: + $ref: "#/components/schemas/Health" + "404": + description: Not implemented; readiness falls back to a TCP port check. + "503": + description: Control API not ready yet. + +components: + schemas: + + Health: + type: object + properties: + status: + type: string + enum: [ok] + description: Present and equal to `ok` when the control API is ready. + required: [status] + + Error: + type: object + properties: + message: + type: string + description: Human-readable explanation. Never interpreted by the TCK. + required: [message] diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py new file mode 100644 index 000000000..bfa3064be --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py @@ -0,0 +1,112 @@ +"""The seam between the scenarios and whatever manipulates the backend.""" + +from __future__ import annotations + +import typing + +__all__ = [ + "BackendControl", + "ConnectionControl", + "UnsupportedControlError", + "unsupported_control", +] + + +class UnsupportedControlError(RuntimeError): + """Raised when a backend cannot perform a control operation. + + It is always a test-configuration bug rather than a provider defect. The + scenarios needing connection control are gated behind + :attr:`Capability.STALE` and :attr:`Capability.UNAVAILABLE_INIT`, so + reaching an unsupported operation means a capability was declared that the + backend cannot back up. The TCK fails loudly on it rather than skipping, + because a silent no-op would report the scenario as passed. + """ + + +@typing.runtime_checkable +class BackendControl(typing.Protocol): + """How the TCK puts the backend under test into the states a scenario needs. + + Step definitions never talk to a backend directly. They talk to this + protocol, which is why the same Gherkin runs unchanged against a + containerised backend driven over HTTP and against a provider manipulated + in-process. Nothing below this line knows about ports, containers or + transports. + + **Which implementation is right for your provider.** If your provider talks + to a backend -- a server, a service, anything out of process -- drive it + over the HTTP control API described in ``control-api.yaml``. That API is the + normative contract for those providers, and it is what makes a conformance + claim portable: another language's TCK drives the same endpoints against the + same stack and must get the same answers. + + Do not write an in-process control that reaches into an external backend + through a side channel -- a test-only admin client, a shared database + handle, a hook inside the provider. It will pass, and it will prove nothing, + because the path it exercised is not the path the contract describes. + + In-process control exists for providers with *no* backend to contract with: + in-memory, environment-variable and file-based providers, where "the + backend" is a data structure in the same process. See + :class:`InProcessControl`. + """ + + def prepare_scenario(self) -> None: + """Bring the backend to the state every scenario starts from. + + Reachable, with flag state at the baseline of the canonical flag set. + Called once before each scenario. This is the TCK's only isolation + mechanism -- scenarios share one backend for the whole suite, and + containers are never restarted between them. + """ + + def change_flag(self) -> None: + """Mutate flag configuration so a conforming provider observes a change. + + Afterwards the provider must resolve a different value for + ``changing-flag``. Which value it changes to is deliberately + unspecified; the suite asserts only that the resolved value differs. + """ + + @property + def description(self) -> str: + """A short description of what is being controlled, for messages a human reads.""" + + +@typing.runtime_checkable +class ConnectionControl(typing.Protocol): + """Implemented by a backend that can be cut off from the provider and restored. + + Separate from :class:`BackendControl` so a backend-less provider cannot + accidentally supply a no-op implementation: not implementing it at all is + the honest answer, and the TCK turns the resulting gap into an explicit, + reported skip. + """ + + def disconnect(self) -> None: + """Make the backend unreachable for the rest of the scenario, without stopping a container.""" + + def reconnect(self) -> None: + """Make the backend reachable again, preserving flag state. + + Preserving flag state is a requirement, not an implementation detail. An + outage must be observable as a change in availability and never as a + change in flag values, or the stale scenario cannot distinguish the two. + """ + + +def unsupported_control(control: BackendControl, operation: str) -> UnsupportedControlError: + """Build the error raised when a backend has no connection to control. + + The message names the fix, because the mistake it reports is always the same + one. + """ + return UnsupportedControlError( + f"{control.description} does not support {operation!r}. This is a " + f"test-configuration bug rather than a provider defect: a scenario needing " + f"connection control ran, so the suite declared Capability.STALE or " + f"Capability.UNAVAILABLE_INIT for a backend that cannot simulate an outage. " + f"Remove those capabilities from TckConfig.capabilities, or supply a " + f"BackendControl that also implements ConnectionControl." + ) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/errors.feature b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/errors.feature new file mode 100644 index 000000000..0346df3da --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/errors.feature @@ -0,0 +1,80 @@ +Feature: Provider error handling + + # Every scenario here asserts the same three-part contract, because all three parts matter and + # providers routinely get one of them wrong: + # + # 1. the code default is returned — an application must keep working, + # 2. the correct error code is reported — an application must be able to tell what went wrong, + # 3. nothing is thrown — an unhandled exception from a flag evaluation is never acceptable. + # + # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. + + Background: + Given a stable provider + + Scenario Outline: Requesting the wrong type returns the code default + # The full non-numeric mismatch matrix. Numeric coercion is a separate question and is covered + # by the @strict-numeric-typing scenarios below, because "is 0.5 an integer?" has a defensible + # wrong answer whereas "is a string a boolean?" does not. + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Examples: a string flag requested as something else + | key | requested | default | + | string-flag | Boolean | false | + | string-flag | Integer | 1 | + | string-flag | Float | 0.1 | + | wrong-flag | Boolean | false | + + Examples: a boolean flag requested as something else + | key | requested | default | + | boolean-flag | String | fallback | + | boolean-flag | Integer | 1 | + | boolean-flag | Float | 0.1 | + + Examples: a numeric flag requested as a non-numeric type + | key | requested | default | + | integer-flag | Boolean | false | + | integer-flag | String | fallback | + | float-flag | Boolean | false | + | float-flag | String | fallback | + + @object + Scenario Outline: Requesting a structured flag as a scalar returns the code default + Given a -flag with key "object-flag" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Examples: + | requested | default | + | Boolean | false | + | String | fallback | + | Integer | 1 | + | Float | 0.1 | + + @strict-numeric-typing + Scenario: A float flag is not silently narrowed to an integer + # 'float-flag' resolves to 0.5. Narrowing that to an integer would lose information + # silently, so it must be reported as a type mismatch rather than rounded. + Given a Integer-flag with key "float-flag" and a default value "1" + When the flag was evaluated with details + Then the resolved details value should be "1" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Scenario: An unknown flag key returns the code default + # 'missing-flag' is deliberately absent from the canonical flag set. + Given a String-flag with key "missing-flag" and a default value "fallback" + When the flag was evaluated with details + Then the resolved details value should be "fallback" + And the reason should be "ERROR" + And the error-code should be "FLAG_NOT_FOUND" + And no exception should have been thrown diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/evaluation.feature b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/evaluation.feature new file mode 100644 index 000000000..e89f174a5 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/evaluation.feature @@ -0,0 +1,59 @@ +Feature: Provider flag evaluation + + # Verifies that a provider maps backend responses onto typed resolution details correctly. + # + # This does NOT test the backend's evaluation logic. Every flag in the canonical set resolves + # to its default variant with no targeting involved, so what is under test is purely the + # provider's mapping of a backend response to a value, a variant and a reason. + # + # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. + + Background: + Given a stable provider + + Scenario Outline: Resolve values with variant and reason + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the variant should be "" + And the reason should be "" + And the error-code should be "" + And no exception should have been thrown + + Examples: + | key | type | default | value | variant | reason | + | boolean-flag | Boolean | false | true | on | STATIC | + | string-flag | String | bye | hi | greeting | STATIC | + | integer-flag | Integer | 1 | 10 | ten | STATIC | + | float-flag | Float | 0.1 | 0.5 | half | STATIC | + + Scenario: An integer flag resolves as an integer + # Paired with the float scenario below and with the narrowing scenario in errors.feature. + # Together they pin down that the two numeric types stay distinct rather than both being + # funnelled through one numeric representation. + Given a Integer-flag with key "integer-flag" and a default value "1" + When the flag was evaluated with details + Then the resolved details value should be "10" + And the error-code should be "" + And no exception should have been thrown + + Scenario: A float flag resolves as a float + Given a Float-flag with key "float-flag" and a default value "0.1" + When the flag was evaluated with details + Then the resolved details value should be "0.5" + And the error-code should be "" + And no exception should have been thrown + + @object + Scenario: Resolve a structured value + Given a Object-flag with key "object-flag" and a default value "{}" + When the flag was evaluated with details + Then the variant should be "template" + And the reason should be "STATIC" + And the error-code should be "" + And no exception should have been thrown + And the resolved object value should contain + | key | type | value | + | showImages | Boolean | true | + | title | String | Check out these pics! | + | imagesPerPage | Integer | 100 | diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/events.feature b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/events.feature new file mode 100644 index 000000000..00e7e5ef6 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/events.feature @@ -0,0 +1,42 @@ +@events +Feature: Provider events + + # Verifies that a provider notices changes in its backend and both signals them and acts on + # them. Signalling alone is not enough: a configuration-change event that is not followed by + # a changed evaluation result is a lie, so each scenario asserts the event AND the behaviour. + # + # Outages here are simulated inside the running stack via the control API. No container is + # ever stopped or restarted — see the invariant in openapi/control-api.yaml. + + Background: + Given a stable provider + + @configuration-change + Scenario: A configuration change is signalled and applied + Given a String-flag with key "changing-flag" and a default value "unset" + And a change event handler + When the flag was evaluated with details + And the resolved value is remembered + And the flag was modified + Then the change event handler should have been executed + And the flag should be part of the event payload + When the flag was evaluated with details + Then the resolved details value should have changed + And no exception should have been thrown + + @stale + Scenario: Losing the backend makes the provider stale, regaining it makes it ready again + Given a ready event handler + And a stale event handler + When a ready event was fired + And the connection is lost + Then the stale event handler should have been executed + And the client should be in stale state + When the connection is restored + Then the ready event handler should have been executed + And the client should be in ready state + + # Deliberately NOT covered here: whether a stale provider keeps serving last-known values + # during the outage. That is caching behaviour, which depends on whether the provider holds a + # local copy of the ruleset, and it belongs behind the @caching capability once those + # scenarios are written. See the "Known gaps" section of the README. diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/lifecycle.feature b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/lifecycle.feature new file mode 100644 index 000000000..256164106 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/lifecycle.feature @@ -0,0 +1,33 @@ +@events +Feature: Provider lifecycle + + # Verifies the two terminal outcomes of provider initialisation: reaching READY against a + # healthy backend, and settling into ERROR against one that cannot be reached. + # + # The failure case matters more than it looks. A provider that blocks forever, or throws out + # of provider registration, takes the host application down with it — so the requirement is + # not merely that initialisation fails, but that it fails observably and promptly. + + Scenario: A provider reaching its backend becomes ready + Given a stable provider + And a ready event handler + Then the ready event handler should have been executed + And the client should be in ready state + + @unavailable + Scenario: A provider that cannot reach its backend reports an error + Given a unavailable provider + And a error event handler + Then the error event handler should have been executed within 10000ms + And the client should be in error state + + @unavailable + Scenario: A provider that cannot reach its backend still returns code defaults + Given a unavailable provider + And a error event handler + And a Boolean-flag with key "boolean-flag" and a default value "false" + Then the error event handler should have been executed within 10000ms + When the flag was evaluated with details + Then the resolved details value should be "false" + And the reason should be "ERROR" + And no exception should have been thrown diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/flag_data/canonical-flags.json b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/flag_data/canonical-flags.json new file mode 100644 index 000000000..343b3ae52 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/flag_data/canonical-flags.json @@ -0,0 +1,82 @@ +{ + "$comment": [ + "The canonical flag set the TCK's feature files assume. A backend under test MUST serve an", + "equivalent set under the configuration named 'default'.", + "", + "Expressed in the flagd flag-definition format because that is the only widely implemented", + "vendor-neutral format today. The format is not what matters — the keys, types, variant", + "names and resolved values are. Seed them however your backend seeds flags.", + "", + "Two things are load-bearing and easy to get wrong:", + " * 'missing-flag' MUST NOT exist. Its absence is what the FLAG_NOT_FOUND scenario tests.", + " * No flag here has targeting rules. Every scenario expects reason STATIC, because the TCK", + " tests the provider's mapping of a response, not the backend's evaluation logic." + ], + "flags": { + "boolean-flag": { + "state": "ENABLED", + "variants": { + "on": true, + "off": false + }, + "defaultVariant": "on" + }, + "string-flag": { + "state": "ENABLED", + "variants": { + "greeting": "hi", + "parting": "bye" + }, + "defaultVariant": "greeting" + }, + "integer-flag": { + "state": "ENABLED", + "variants": { + "one": 1, + "ten": 10 + }, + "defaultVariant": "ten" + }, + "float-flag": { + "state": "ENABLED", + "variants": { + "tenth": 0.1, + "half": 0.5 + }, + "defaultVariant": "half" + }, + "object-flag": { + "state": "ENABLED", + "variants": { + "empty": {}, + "template": { + "showImages": true, + "title": "Check out these pics!", + "imagesPerPage": 100 + } + }, + "defaultVariant": "template" + }, + "wrong-flag": { + "$comment": "A string flag, evaluated as a boolean by the TYPE_MISMATCH scenario.", + "state": "ENABLED", + "variants": { + "one": "uno", + "two": "dos" + }, + "defaultVariant": "one" + }, + "changing-flag": { + "$comment": [ + "The flag POST /change mutates. The TCK asserts only that its resolved value differs", + "after the change, so which of the two variants you start from does not matter." + ], + "state": "ENABLED", + "variants": { + "foo": "foo", + "bar": "bar" + }, + "defaultVariant": "foo" + } + } +} diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py new file mode 100644 index 000000000..1d69254c6 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py @@ -0,0 +1,112 @@ +"""In-process backend control, for providers with no backend at all.""" + +from __future__ import annotations + +from openfeature.provider import FeatureProvider + +from .provider import ( + CHANGING_FLAG_KEY, + ControllableInMemoryProvider, + canonical_flag_set, + changing_flag, +) + +__all__ = ["InProcessControl"] + +_BASELINE = "foo" +_CHANGED = "bar" + + +class InProcessControl: + """Manipulates an in-process provider directly, with no backend and no HTTP. + + This exists so providers with nothing to connect to -- in-memory, + environment-variable and file-based providers -- can run the TCK. For those, + "the backend" is a data structure in the same process: seeding flags is + building a mapping, and changing one is an update on the live provider, so + the event the suite awaits is the provider's own + ``PROVIDER_CONFIGURATION_CHANGED`` rather than one the TCK synthesised. + + **This is not a shortcut for providers that do have a backend.** Reaching + into an external backend from inside the test process -- a test-only admin + client, a shared database handle, a hook in the provider -- produces a suite + that passes while proving nothing, because the path it exercised is not the + path the contract describes. Those providers drive the HTTP control API + instead. + + **Connection control.** :class:`InProcessControl` deliberately does not + implement :class:`~.control.ConnectionControl`. An in-memory provider has no + connection to lose, and pretending otherwise with a no-op would report the + ``@stale`` scenarios as passed. A suite using it leaves + :attr:`Capability.STALE` and :attr:`Capability.UNAVAILABLE_INIT` undeclared, + and those scenarios are skipped with the reason reported. + + **Ownership of the provider.** This type both seeds the flags and creates + the provider serving them, because in-process they are the same object: + :meth:`change_flag` has to reach the live instance to emit an event from it. + A suite therefore wires both through one control:: + + control = InProcessControl() + TckConfig( + name="in-memory", + control=control, + new_provider=control.new_provider, + capabilities={Capability.EVENTS, Capability.CONFIGURATION_CHANGE}, + ) + """ + + def __init__(self) -> None: + self._current: ControllableInMemoryProvider | None = None + self._changing_variant = _BASELINE + + @property + def description(self) -> str: + return "in-process control of an in-memory provider" + + def new_provider(self) -> FeatureProvider: + """Create the provider for the scenario about to run, at the baseline. + + Each call returns a fresh instance over a fresh copy of the canonical + flag set, which is what makes :meth:`prepare_scenario` nothing more than + dropping the previous reference. + """ + self._changing_variant = _BASELINE + self._current = ControllableInMemoryProvider(canonical_flag_set()) + return self._current + + def prepare_scenario(self) -> None: + """Drop the previous scenario's provider. + + That is the whole reset: the flag set is rebuilt per provider, so the + :meth:`new_provider` call that follows starts from an untouched + baseline. Clearing the reference rather than leaving it dangling means a + scenario that changes flags without creating a provider fails with a + clear message instead of mutating one that has already been shut down. + """ + self._current = None + + def change_flag(self) -> None: + """Flip ``changing-flag`` between its two variants on the live provider. + + The event the suite awaits is therefore the provider's own + ``PROVIDER_CONFIGURATION_CHANGED``, carrying ``changing-flag`` in + ``flags_changed``, and not a signal the TCK synthesised. + + Alternating rather than assigning a fixed variant keeps repeated calls + within one scenario meaningful; the suite asserts that the resolved + value differs, not what it became. + """ + if self._current is None: + msg = ( + "No in-memory provider exists for this scenario. In-process control " + "manipulates the provider itself, so the scenario must create one -- " + 'with "Given a stable provider" -- before any step that changes flag state.' + ) + raise RuntimeError(msg) + + self._changing_variant = ( + _BASELINE if self._changing_variant == _CHANGED else _CHANGED + ) + self._current.update_flag( + CHANGING_FLAG_KEY, changing_flag(self._changing_variant) + ) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py new file mode 100644 index 000000000..239c41acd --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py @@ -0,0 +1,104 @@ +"""The pytest plugin: capability gating, scenario state, and the shared step vocabulary. + +Registered through the ``pytest11`` entry point, so installing this package is +all it takes for the step definitions to be available. pytest-bdd resolves steps +through the fixture system and fixtures from an installed plugin are visible to +every test, which is what keeps an adoption down to one fixture and one call to +:func:`tck_scenarios`. +""" + +from __future__ import annotations + +import typing + +import pytest + +from openfeature import api + +from .capability import Capability, capability_for_marker +from .config import TckConfig +from .state import TckState + +# The step modules are registered as plugins in their own right, not merely +# imported. pytest-bdd's decorators inject a generated fixture name into the +# *defining* module's namespace, so a step is only visible to pytest once the +# module defining it is a registered plugin -- importing it here would run the +# decorators but leave those fixtures where pytest never looks. +pytest_plugins = [ + "openfeature.contrib.tools.provider_tck.steps.provider_steps", + "openfeature.contrib.tools.provider_tck.steps.flag_steps", + "openfeature.contrib.tools.provider_tck.steps.event_steps", +] + +def pytest_configure(config: pytest.Config) -> None: + """Register the capability tags as markers. + + pytest-bdd turns every Gherkin tag into a marker with + ``getattr(pytest.mark, tag)`` without registering it, which raises + ``PytestUnknownMarkWarning`` for each one -- noise at best, and a hard + failure in a project configured with ``-W error``. + """ + for capability in Capability: + config.addinivalue_line( + "markers", + f"{capability.value}: OpenFeature provider TCK capability {capability.tag}", + ) + + +@pytest.fixture +def tck_state(tck_config: TckConfig) -> typing.Iterator[TckState]: + """Per-scenario state, carried between step definitions.""" + # Resetting here rather than in an autouse fixture ties the reset to the + # scenarios that actually use the TCK, and guarantees it happens after the + # capability gate has had its say -- a skipped scenario never touches the + # backend. + tck_config.control.prepare_scenario() + state = TckState(config=tck_config) + yield state + state.teardown() + + +@pytest.fixture(autouse=True) +def _tck_capability_gate(request: pytest.FixtureRequest) -> None: + """Skip a scenario whose capability the provider did not declare. + + ``pytest.skip`` here reports the scenario as skipped **with the reason**, + which is exactly what the specification asks a TCK implementation to do. + Nothing about it can be mistaken for a pass. + + The gate keys off the node's markers rather than its requested fixtures. + pytest-bdd resolves a step's fixtures lazily, as each step runs, so + ``tck_config`` is not in ``request.fixturenames`` when this autouse fixture + is set up -- guarding on that silently disabled the gate and let + ``@unavailable`` scenarios run against a config that never declared it. + + Checking markers first also means the gate costs nothing, and instantiates + nothing, for tests that are not TCK scenarios. + """ + gated = [ + capability + for marker in request.node.iter_markers() + if (capability := capability_for_marker(marker.name)) is not None + ] + if not gated: + return + + try: + config: TckConfig = request.getfixturevalue("tck_config") + except pytest.FixtureLookupError: + return + + for capability in gated: + if not config.declares(capability): + pytest.skip( + f"provider does not declare capability {capability.tag}. " + f"Declared: {' '.join(config.sorted_capabilities) or '(none)'}" + ) + + +@pytest.fixture(scope="session", autouse=True) +def _tck_release_providers() -> typing.Iterator[None]: + """Shut down whatever the suite registered once it is over.""" + yield + api.shutdown() + api.clear_providers() diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py new file mode 100644 index 000000000..5b1c9faa4 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py @@ -0,0 +1,131 @@ +"""An in-memory provider that can be reconfigured at runtime, and the canonical flag set.""" + +from __future__ import annotations + +import typing + +from openfeature.event import ProviderEventDetails +from openfeature.provider.in_memory_provider import ( + FlagStorage, + InMemoryFlag, + InMemoryProvider, +) + +__all__ = [ + "CHANGING_FLAG_KEY", + "ControllableInMemoryProvider", + "canonical_flag_set", + "changing_flag", +] + +CHANGING_FLAG_KEY = "changing-flag" +"""The flag :meth:`BackendControl.change_flag` mutates.""" + +_CHANGING_BASELINE = "foo" +_CHANGING_CHANGED = "bar" + + +class ControllableInMemoryProvider(InMemoryProvider): + """An in-memory provider whose flag set can be replaced at runtime. + + **Why this exists.** `Appendix A`_ of the specification requires an SDK's + in-memory provider to "support a means of updating the ``flag set``, + resulting in the emission of ``PROVIDER_CONFIGURATION_CHANGED`` events". The + Python SDK's :class:`~openfeature.provider.in_memory_provider.InMemoryProvider` + has no such method: it copies the flag mapping in its constructor and never + exposes a way to change it. + + Only half the machinery is missing, which is what makes this a small class + rather than a reimplementation. :class:`~openfeature.provider.AbstractProvider` + already supplies ``emit_provider_configuration_changed``, and the registry + already attaches the emitter, so all that is needed is a method that swaps + the mapping and emits. Everything about *resolution* -- variants, reasons, + ``FLAG_NOT_FOUND`` -- is still the SDK's. + + That makes this an honest reference for what the SDK's provider should grow, + rather than a competing implementation that could drift from it. + + .. _Appendix A: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md + """ + + def update_flags(self, flags: FlagStorage) -> None: + """Replace the whole flag set and emit a configuration-change event. + + The event names the union of the previous and new keys, which is what + Appendix A asks for: a consumer caching evaluations needs to know + everything that might have changed, and a key that disappeared has + changed as much as one that was added. + """ + changed = sorted(set(self._flags) | set(flags)) + self._flags = dict(flags) + self.emit_provider_configuration_changed( + ProviderEventDetails( + flags_changed=changed, message="flag configuration changed" + ) + ) + + def update_flag(self, key: str, flag: InMemoryFlag[typing.Any]) -> None: + """Replace a single flag and emit a configuration-change event naming it.""" + updated = dict(self._flags) + updated[key] = flag + self._flags = updated + self.emit_provider_configuration_changed( + ProviderEventDetails( + flags_changed=[key], message="flag configuration changed" + ) + ) + + def flag(self, key: str) -> InMemoryFlag[typing.Any] | None: + """Return the flag currently registered under ``key``.""" + return self._flags.get(key) + + +def changing_flag(default_variant: str) -> InMemoryFlag[str]: + return InMemoryFlag( + default_variant=default_variant, + variants={_CHANGING_BASELINE: _CHANGING_BASELINE, _CHANGING_CHANGED: _CHANGING_CHANGED}, + ) + + +def canonical_flag_set() -> FlagStorage: + """Return the canonical flag set as SDK in-memory flags. + + Mirrors ``flag_data/canonical-flags.json`` entry for entry. Two properties + of that file are load-bearing and hold here too: + + * ``missing-flag`` is absent, which is what the ``FLAG_NOT_FOUND`` scenario + tests. Adding it turns that scenario green for the wrong reason. + * no flag carries a ``context_evaluator``, so every evaluation reports reason + ``STATIC`` -- the TCK tests a provider's mapping of a response, not a + backend's evaluation logic. + """ + return { + "boolean-flag": InMemoryFlag( + default_variant="on", variants={"on": True, "off": False} + ), + "string-flag": InMemoryFlag( + default_variant="greeting", variants={"greeting": "hi", "parting": "bye"} + ), + "integer-flag": InMemoryFlag( + default_variant="ten", variants={"one": 1, "ten": 10} + ), + "float-flag": InMemoryFlag( + default_variant="half", variants={"tenth": 0.1, "half": 0.5} + ), + "object-flag": InMemoryFlag( + default_variant="template", + variants={ + "empty": {}, + "template": { + "showImages": True, + "title": "Check out these pics!", + "imagesPerPage": 100, + }, + }, + ), + # A string flag, evaluated as a boolean by the TYPE_MISMATCH scenario. + "wrong-flag": InMemoryFlag( + default_variant="one", variants={"one": "uno", "two": "dos"} + ), + CHANGING_FLAG_KEY: changing_flag(_CHANGING_BASELINE), + } diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py new file mode 100644 index 000000000..71ea41508 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py @@ -0,0 +1,146 @@ +"""Per-scenario state: what a scenario accumulates, and how it observes events. + +Separate from :mod:`plugin` so the step modules can import these types at the top +level. The step modules are loaded by the plugin as plugins in their own right, +and a step importing from the plugin module that loads it reads like a cycle even +where it is not one. +""" + +from __future__ import annotations + +import queue +import typing +from dataclasses import dataclass, field + +from openfeature.client import OpenFeatureClient +from openfeature.event import EventDetails, ProviderEvent +from openfeature.flag_evaluation import FlagType + +from .config import TckConfig + +__all__ = ["EvaluationRecord", "EventRecorder", "TckState"] + + +@dataclass +class EvaluationRecord: + """The outcome of one flag evaluation, flattened across the five typed calls.""" + + value: typing.Any = None + variant: str | None = None + reason: str | None = None + error_code: str | None = None + error_message: str | None = None + raised: BaseException | None = None + """The exception the call raised, if any. + + In Python an errored evaluation returns the code default in the details + rather than raising, so this stays ``None`` on the error paths the suite + exercises. It is what "no exception should have been thrown" asserts. + """ + + +class EventRecorder: + """Captures the events of one type, in order, so a scenario consumes them one at a time. + + Consuming rather than merely observing is what makes the stale scenario + work: it awaits a ``PROVIDER_READY`` at the start and a second, different + ``PROVIDER_READY`` once the backend is back, and a recorder that only + remembered "ready has fired at some point" would report the second assertion + as satisfied by the first event. + + A queue rather than a list because a provider with a background thread -- + anything with a real backend -- delivers events from that thread while the + scenario waits on the main one. + """ + + def __init__(self, client: OpenFeatureClient, event: ProviderEvent) -> None: + self.event = event + self._client = client + self._events: queue.Queue[EventDetails] = queue.Queue() + self.last: EventDetails | None = None + + # The SDK replays a matching event on registration when the provider is + # already in the corresponding state, so a handler added after the + # provider became ready still observes its PROVIDER_READY. That is what + # lets the feature files register handlers after "Given a stable + # provider" without racing it. + client.add_handler(event, self._on_event) + + def _on_event(self, details: EventDetails) -> None: + self._events.put(details) + + def await_event(self, timeout: float) -> EventDetails: + """Consume the next event of this recorder's type.""" + try: + details = self._events.get(timeout=timeout) + except queue.Empty: + msg = ( + f"timed out after {timeout}s waiting for a {self.event.value} event. " + f"If the provider is simply slower than this to notice, raise " + f"TckConfig.event_timeout rather than treating it as a failure" + ) + raise AssertionError(msg) from None + self.last = details + return details + + def detach(self) -> None: + self._client.remove_handler(self.event, self._on_event) + + +@dataclass +class TckState: + """Everything one scenario accumulates.""" + + config: TckConfig + client: OpenFeatureClient | None = None + flag_key: str | None = None + flag_type: FlagType | None = None + default_value: typing.Any = None + last: EvaluationRecord | None = None + remembered: typing.Any = None + has_memory: bool = False + recorders: dict[ProviderEvent, EventRecorder] = field(default_factory=dict) + + def require_client(self) -> OpenFeatureClient: + if self.client is None: + msg = ( + "no provider has been registered in this scenario: a " + '"Given a stable provider" or "Given a unavailable provider" step ' + "must come first" + ) + raise AssertionError(msg) + return self.client + + def require_flag(self) -> tuple[str, FlagType, typing.Any]: + if self.flag_key is None or self.flag_type is None: + msg = ( + "no flag has been declared in this scenario: a " + '"Given a -flag with key ... and a default value ..." step ' + "must come first" + ) + raise AssertionError(msg) + return self.flag_key, self.flag_type, self.default_value + + def require_evaluation(self) -> EvaluationRecord: + if self.last is None: + msg = ( + "no flag has been evaluated in this scenario: a " + '"When the flag was evaluated with details" step must come first' + ) + raise AssertionError(msg) + return self.last + + def require_recorder(self, event: ProviderEvent) -> EventRecorder: + recorder = self.recorders.get(event) + if recorder is None: + msg = ( + f"no handler was registered for {event.value} in this scenario: a " + '"Given a event handler" step must come first' + ) + raise AssertionError(msg) + return recorder + + def teardown(self) -> None: + for recorder in self.recorders.values(): + recorder.detach() + self.recorders.clear() diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/__init__.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/__init__.py new file mode 100644 index 000000000..6581ee86f --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/__init__.py @@ -0,0 +1,11 @@ +"""The shared step vocabulary. + +Each module here is registered as a pytest plugin by the TCK's own plugin, which +is what makes the steps visible: pytest-bdd's decorators inject a generated +fixture name into the *defining* module's namespace, so a step only reaches +pytest once its module is a registered plugin. + +Deliberately empty of imports. Pulling the submodules in here would import them +before pytest loads them as plugins, and pytest cannot rewrite assertions in a +module that is already imported. +""" diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py new file mode 100644 index 000000000..47a37da8f --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py @@ -0,0 +1,167 @@ +"""Steps covering provider events, connection loss and client status.""" + +from __future__ import annotations + +from pytest_bdd import given, parsers, then, when + +from openfeature.event import ProviderEvent +from openfeature.provider import ProviderStatus + +from ..control import ConnectionControl, unsupported_control +from ..state import EventRecorder, TckState + +__all__ = [ + "an_event_handler", + "an_event_was_fired", + "the_client_should_be_in_state", + "the_connection_is_lost", + "the_connection_is_restored", + "the_event_handler_should_have_been_executed", + "the_event_handler_should_have_been_executed_within", + "the_flag_should_be_part_of_the_event_payload", +] + +_EVENT_BY_NAME: dict[str, ProviderEvent] = { + "ready": ProviderEvent.PROVIDER_READY, + "stale": ProviderEvent.PROVIDER_STALE, + "error": ProviderEvent.PROVIDER_ERROR, + "change": ProviderEvent.PROVIDER_CONFIGURATION_CHANGED, +} + +_STATUS_BY_NAME: dict[str, ProviderStatus] = { + "ready": ProviderStatus.READY, + "stale": ProviderStatus.STALE, + "error": ProviderStatus.ERROR, +} + + +def _event(name: str) -> ProviderEvent: + try: + return _EVENT_BY_NAME[name] + except KeyError: + msg = f"unknown event kind {name!r}" + raise AssertionError(msg) from None + + +@given(parsers.re(r"^an? (?Pready|stale|error|change) event handler$")) +def an_event_handler(tck_state: TckState, kind: str) -> None: + """Attach a recorder for one event type. + + Handlers are attached after the provider is registered, which the SDK + handles by replaying a matching event on registration when the provider is + already in the corresponding state. That is why "Given a stable provider" + followed by "And a ready event handler" is not a race. + """ + event = _event(kind) + if event in tck_state.recorders: + return + client = tck_state.require_client() + tck_state.recorders[event] = EventRecorder(client, event) + + +@when(parsers.re(r"^a (?Pready|stale|error|change) event was fired$")) +def an_event_was_fired(tck_state: TckState, kind: str) -> None: + """Consume an event, so a later assertion observes the next one rather than this. + + The stale scenario depends on it: it consumes the initial ``PROVIDER_READY`` + here and then asserts a second, distinct one once the backend is back. + """ + recorder = tck_state.require_recorder(_event(kind)) + recorder.await_event(tck_state.config.event_timeout) + + +@then( + parsers.re( + r"^the (?Pready|stale|error|change) event handler should have been executed$" + ) +) +def the_event_handler_should_have_been_executed(tck_state: TckState, kind: str) -> None: + recorder = tck_state.require_recorder(_event(kind)) + recorder.await_event(tck_state.config.event_timeout) + + +@then( + parsers.re( + r"^the (?Pready|stale|error|change) event handler should have been " + r"executed within (?P\d+)ms$" + ) +) +def the_event_handler_should_have_been_executed_within( + tck_state: TckState, kind: str, millis: str +) -> None: + """Bound the wait explicitly. + + The scenarios using this assert promptness, not merely eventual arrival: a + provider that cannot reach its backend has to report that fact quickly, + because an application blocked on provider registration is down. The bound + therefore overrides ``event_timeout`` rather than being clamped by it. + """ + recorder = tck_state.require_recorder(_event(kind)) + recorder.await_event(int(millis) / 1000.0) + + +@then("the flag should be part of the event payload") +def the_flag_should_be_part_of_the_event_payload(tck_state: TckState) -> None: + """Assert the configuration-change event named the flag that changed. + + Naming the changed flags is what makes the event actionable: a consumer + caching evaluations needs to know what to invalidate, and an event carrying + no keys forces it to invalidate everything. + """ + key, _flag_type, _default = tck_state.require_flag() + recorder = tck_state.require_recorder(ProviderEvent.PROVIDER_CONFIGURATION_CHANGED) + + if recorder.last is None: + msg = ( + "no configuration-change event has been consumed in this scenario: a " + '"the change event handler should have been executed" step must come first' + ) + raise AssertionError(msg) + + changed = recorder.last.flags_changed or [] + if key in changed: + return + + if not changed: + msg = ( + f"the configuration-change event carried no changed flags, expected it to " + f"name {key!r}" + ) + else: + msg = ( + f"the configuration-change event named {changed}, expected it to include {key!r}" + ) + raise AssertionError(msg) + + +def _connection_control(tck_state: TckState, operation: str) -> ConnectionControl: + control = tck_state.config.control + if not isinstance(control, ConnectionControl): + raise unsupported_control(control, operation) + return control + + +@when("the connection is lost") +def the_connection_is_lost(tck_state: TckState) -> None: + _connection_control(tck_state, "disconnect").disconnect() + + +@when("the connection is restored") +def the_connection_is_restored(tck_state: TckState) -> None: + _connection_control(tck_state, "reconnect").reconnect() + + +@then(parsers.re(r"^the client should be in (?Pready|stale|error) state$")) +def the_client_should_be_in_state(tck_state: TckState, name: str) -> None: + """Assert the provider status the client reports. + + Checked after the corresponding event has been consumed, and the SDK writes + provider status before running handlers, so no polling is needed: if the + event arrived, the status is already current. + """ + client = tck_state.require_client() + expected = _STATUS_BY_NAME[name] + actual = client.get_provider_status() + if actual != expected: + msg = f"client reports status {actual}, expected {expected}" + raise AssertionError(msg) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py new file mode 100644 index 000000000..f45b8dbff --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py @@ -0,0 +1,238 @@ +"""Steps that declare, evaluate and assert flags.""" + +from __future__ import annotations + +import typing +from collections.abc import Callable + +from pytest_bdd import given, parsers, then, when + +from openfeature.flag_evaluation import FlagType + +from ..state import EvaluationRecord, TckState +from ..values import describe, parse_flag_type, parse_value, values_equal + +__all__ = [ + "a_flag_with_key_and_default", + "no_exception_should_have_been_thrown", + "the_error_code_should_be", + "the_flag_was_evaluated_with_details", + "the_flag_was_modified", + "the_reason_should_be", + "the_resolved_object_value_should_contain", + "the_resolved_value_is_remembered", + "the_resolved_value_should_be", + "the_resolved_value_should_have_changed", + "the_variant_should_be", +] + + +@given( + parsers.re( + r'^an? (?P[A-Za-z]+)-flag with key "(?P[^"]*)" ' + r'and a default value "(?P[^"]*)"$' + ) +) +def a_flag_with_key_and_default( + tck_state: TckState, flag_type: str, key: str, default: str +) -> None: + """Declare the flag the scenario is about, and the type it is requested as. + + The two are independent on purpose: most of ``errors.feature`` asks for a + flag as a type it is not. + """ + parsed_type = parse_flag_type(flag_type) + tck_state.flag_key = key + tck_state.flag_type = parsed_type + tck_state.default_value = parse_value(parsed_type, default) + + +@when("the flag was evaluated with details") +def the_flag_was_evaluated_with_details(tck_state: TckState) -> None: + """Resolve the declared flag through the typed client call matching its type.""" + client = tck_state.require_client() + key, flag_type, default = tck_state.require_flag() + + # Annotated explicitly: the five typed getters have different signatures, so + # an unannotated mapping infers a value type mypy will not let us call. + calls: dict[FlagType, Callable[[str, typing.Any], typing.Any]] = { + FlagType.BOOLEAN: client.get_boolean_details, + FlagType.STRING: client.get_string_details, + FlagType.INTEGER: client.get_integer_details, + FlagType.FLOAT: client.get_float_details, + FlagType.OBJECT: client.get_object_details, + } + + record = EvaluationRecord() + try: + details = calls[flag_type](key, default) + except BaseException as exc: # recorded here, asserted on by its own step + record.raised = exc + record.value = default + else: + record.value = details.value + record.variant = details.variant + record.reason = str(details.reason) if details.reason is not None else None + record.error_code = ( + details.error_code.value if details.error_code is not None else None + ) + record.error_message = details.error_message + + tck_state.last = record + + +@then(parsers.re(r'^the resolved details value should be "(?P[^"]*)"$')) +def the_resolved_value_should_be(tck_state: TckState, expected: str) -> None: + _key, flag_type, _default = tck_state.require_flag() + record = tck_state.require_evaluation() + wanted = parse_value(flag_type, expected) + + if not values_equal(wanted, record.value): + detail = f" (the client also reported: {record.error_message})" if record.error_message else "" + msg = ( + f"flag {tck_state.flag_key!r} resolved to {describe(record.value)}, " + f"expected {describe(wanted)}{detail}" + ) + raise AssertionError(msg) + + +@then(parsers.re(r'^the variant should be "(?P[^"]*)"$')) +def the_variant_should_be(tck_state: TckState, expected: str) -> None: + record = tck_state.require_evaluation() + if record.variant != expected: + msg = ( + f"variant was {record.variant!r}, expected {expected!r}. A variant that " + f"does not survive the trip from the backend is one of the easiest parts " + f"of the contract to drop" + ) + raise AssertionError(msg) + + +@then(parsers.re(r'^the reason should be "(?P[^"]*)"$')) +def the_reason_should_be(tck_state: TckState, expected: str) -> None: + record = tck_state.require_evaluation() + if record.reason != expected: + msg = f"reason was {record.reason!r}, expected {expected!r}" + raise AssertionError(msg) + + +@then(parsers.re(r'^the error-code should be "(?P[^"]*)"$')) +def the_error_code_should_be(tck_state: TckState, expected: str) -> None: + """Assert the reported error code, where the empty string means none at all. + + The empty case matters as much as the populated ones. A provider that + reports a plausible value with no error code is the failure mode the suite + is most concerned with, because the application has no way to notice. + """ + record = tck_state.require_evaluation() + actual = record.error_code or "" + + if actual == expected: + return + + if expected == "": + msg = f"error-code was {actual!r}, expected none" + elif actual == "": + msg = ( + f"no error-code was reported, expected {expected!r}. Returning a value " + f"without an error code leaves the application unable to tell that " + f"anything went wrong" + ) + else: + msg = f"error-code was {actual!r}, expected {expected!r}" + raise AssertionError(msg) + + +@then("no exception should have been thrown") +def no_exception_should_have_been_thrown(tck_state: TckState) -> None: + """Assert the evaluation returned rather than raised. + + In Python an errored evaluation returns the code default in the details and + does not raise, so this holds on the error paths too. A provider that raises + instead takes the calling application down with it, which is what the + feature files forbid. + """ + record = tck_state.require_evaluation() + if record.raised is not None: + msg = ( + f"the evaluation raised {record.raised!r}. A flag evaluation must always " + f"return a value and an error code, never raise" + ) + raise AssertionError(msg) + + +@then("the resolved object value should contain") +def the_resolved_object_value_should_contain( + tck_state: TckState, datatable: list[list[str]] +) -> None: + """Assert members of a structured value, each with its own expected type.""" + record = tck_state.require_evaluation() + header, *rows = datatable + + if header != ["key", "type", "value"]: + msg = f"expected a data table with columns key, type, value; got {header}" + raise AssertionError(msg) + + if not isinstance(record.value, dict): + msg = ( + f"resolved object value is {describe(record.value)}, which has no members " + f"to check" + ) + raise AssertionError(msg) + + for key, raw_type, raw_value in rows: + wanted = parse_value(parse_flag_type(raw_type), raw_value) + if key not in record.value: + msg = f"resolved object value has no member {key!r}" + raise AssertionError(msg) + actual = record.value[key] + if not values_equal(wanted, actual): + msg = ( + f"object member {key!r} was {describe(actual)}, expected {describe(wanted)}" + ) + raise AssertionError(msg) + + +@when("the resolved value is remembered") +def the_resolved_value_is_remembered(tck_state: TckState) -> None: + """Store the current value so a later step can assert it changed.""" + record = tck_state.require_evaluation() + tck_state.remembered = record.value + tck_state.has_memory = True + + +@then("the resolved details value should have changed") +def the_resolved_value_should_have_changed(tck_state: TckState) -> None: + """Assert that re-evaluation produced a different value. + + This is the half of the configuration-change contract providers actually get + wrong. Emitting ``PROVIDER_CONFIGURATION_CHANGED`` and then continuing to + resolve the old value is worse than emitting nothing, because the + application acted on a signal that was not true. + """ + record = tck_state.require_evaluation() + if not tck_state.has_memory: + msg = ( + "no value was remembered in this scenario: a " + '"the resolved value is remembered" step must come first' + ) + raise AssertionError(msg) + + if values_equal(tck_state.remembered, record.value): + msg = ( + f"the resolved value is still {describe(record.value)} after the " + f"configuration changed. The change was signalled but not applied, so the " + f"event told the application something untrue" + ) + raise AssertionError(msg) + + +@when("the flag was modified") +def the_flag_was_modified(tck_state: TckState) -> None: + """Change flag configuration on the backend.""" + control = tck_state.config.control + try: + control.change_flag() + except Exception as exc: + msg = f"could not change flag configuration on {control.description}: {exc}" + raise AssertionError(msg) from exc diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py new file mode 100644 index 000000000..057b24842 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py @@ -0,0 +1,81 @@ +"""Steps that put a provider under test.""" + +from __future__ import annotations + +import contextlib + +from pytest_bdd import given, parsers + +from openfeature import api + +from ..state import TckState + +__all__ = ["a_stable_provider", "an_unavailable_provider"] + + +@given(parsers.re(r"^an? stable provider$")) +def a_stable_provider(tck_state: TckState) -> None: + """Register the provider under test against the running, seeded backend. + + ``api.set_provider`` initialises synchronously and dispatches + ``PROVIDER_READY``, so by the time this step returns the provider is ready + and every scenario that follows can assume it. A suite that started + evaluating before that would report races in the TCK as defects in the + provider. + """ + config = tck_state.config + provider = config.new_provider() + if provider is None: + msg = "TckConfig.new_provider returned None" + raise AssertionError(msg) + + try: + api.set_provider(provider, config.domain) + except Exception as exc: + msg = ( + f"registering the provider raised {exc!r}. The backend is up and seeded " + f"at this point, so this is a genuine initialisation failure rather than " + f"the unavailable-backend case" + ) + raise AssertionError(msg) from exc + + tck_state.client = api.get_client(config.domain) + + +@given(parsers.re(r"^an? unavailable provider$")) +def an_unavailable_provider(tck_state: TckState) -> None: + """Register a provider pointed at a backend that does not exist. + + Neither a failed initialisation nor a raised exception during registration + is a failure here: what the contract requires is that the provider settles + into an observable error state promptly, which the scenario asserts through + the event and the client status. The SDK's registry already converts a + raising ``initialize`` into ``PROVIDER_ERROR``, so registration itself is + expected to return normally -- but a provider that raises anyway must not + take the scenario down with it, which is why this is caught rather than + propagated. + """ + config = tck_state.config + + if config.new_unavailable_provider is None: + msg = ( + "TckConfig.new_unavailable_provider is None but an @unavailable scenario " + "ran. This is a test-configuration bug rather than a provider defect: the " + "suite declared Capability.UNAVAILABLE_INIT without supplying a provider " + "that cannot reach its backend. Remove that capability, or supply the factory" + ) + raise AssertionError(msg) + + provider = config.new_unavailable_provider() + if provider is None: + msg = "TckConfig.new_unavailable_provider returned None" + raise AssertionError(msg) + + # A raising initialize is already converted to PROVIDER_ERROR by the SDK's + # registry, so this is belt and braces: a provider that raises anyway must + # not take the scenario down with it, because the contract is about the + # observable error state rather than about how registration returned. + with contextlib.suppress(Exception): + api.set_provider(provider, config.domain) + + tck_state.client = api.get_client(config.domain) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py new file mode 100644 index 000000000..13dbe696e --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py @@ -0,0 +1,121 @@ +"""Turning Gherkin strings into typed values, and comparing them with what a provider resolved.""" + +from __future__ import annotations + +import json +import typing + +from openfeature.flag_evaluation import FlagType + +__all__ = ["describe", "parse_flag_type", "parse_value", "values_equal"] + +_BY_NAME: dict[str, FlagType] = { + "boolean": FlagType.BOOLEAN, + "string": FlagType.STRING, + "integer": FlagType.INTEGER, + "float": FlagType.FLOAT, + "object": FlagType.OBJECT, +} + + +def parse_flag_type(raw: str) -> FlagType: + """Resolve the type named in a scenario, case-insensitively.""" + try: + return _BY_NAME[raw.strip().lower()] + except KeyError: + names = ", ".join(sorted(n.capitalize() for n in _BY_NAME)) + msg = f"unknown flag type {raw!r}: expected one of {names}" + raise ValueError(msg) from None + + +def _parse_bool(raw: str) -> bool: + lowered = raw.strip().lower() + if lowered in {"true", "t", "yes", "1"}: + return True + if lowered in {"false", "f", "no", "0"}: + return False + msg = f"{raw!r} is not a boolean" + raise ValueError(msg) + + +def parse_value(flag_type: FlagType, raw: str) -> typing.Any: + """Convert a value written in a scenario into the type the API uses. + + Everything in Gherkin is a string, so this is where ``"0.5"`` becomes a + float and ``"{}"`` becomes an empty object. Parsing per declared type rather + than guessing is what keeps the integer and float scenarios + distinguishable: ``"1"`` is an ``int`` in an Integer scenario and a ``float`` + in a Float one. + """ + if flag_type is FlagType.BOOLEAN: + return _parse_bool(raw) + if flag_type is FlagType.STRING: + return raw + if flag_type is FlagType.INTEGER: + return int(raw) + if flag_type is FlagType.FLOAT: + return float(raw) + if flag_type is FlagType.OBJECT: + # Gherkin escapes quotes in table cells; pytest-bdd keeps the backslash, + # so strip it before handing the text to json. + return json.loads(raw.replace('\\"', '"')) + msg = f"unknown flag type {flag_type!r}" + raise ValueError(msg) + + +def _as_number(value: typing.Any) -> float | None: + """Return a numeric value as a float, or None if it is not numeric. + + Booleans are deliberately excluded. Python makes ``bool`` a subclass of + ``int``, so an unguarded numeric comparison would quietly report ``True`` and + ``1`` as equal -- which is the exact confusion several of these scenarios + exist to detect. + """ + if isinstance(value, bool): + return None + if isinstance(value, (int, float)): + return float(value) + return None + + +def values_equal(expected: typing.Any, actual: typing.Any) -> bool: + """Compare an expected value from a scenario with what a provider resolved. + + Numbers are compared numerically rather than by Python type. A provider that + deserialises its backend's JSON hands back ``float`` for every number, so the + ``100`` inside ``object-flag`` arrives as ``100.0`` from one provider and + ``100`` from another while both are correct. Type distinctness is asserted + where it belongs -- by requesting a flag as a specific type and checking the + error code -- not by accident of how a number was decoded. + """ + # A boolean only ever equals a boolean. Without this, Python's bool-is-an-int + # rule would make True == 1 and quietly satisfy the scenario that exists to + # catch exactly that confusion. + if isinstance(expected, bool) or isinstance(actual, bool): + return isinstance(expected, bool) and isinstance(actual, bool) and expected == actual + + expected_number = _as_number(expected) + if expected_number is not None: + actual_number = _as_number(actual) + return actual_number is not None and expected_number == actual_number + + if isinstance(expected, dict) and isinstance(actual, dict): + if set(expected) != set(actual): + return False + return all(values_equal(v, actual[k]) for k, v in expected.items()) + + if isinstance(expected, list) and isinstance(actual, list): + return len(expected) == len(actual) and all( + values_equal(e, a) for e, a in zip(expected, actual, strict=True) + ) + + return bool(expected == actual) + + +def describe(value: typing.Any) -> str: + """Render a value for a failure message, including its type. + + "expected 100 but got 100" is the single most confusing failure a + cross-language conformance suite can produce. + """ + return f"{value!r} ({type(value).__name__})" diff --git a/tools/openfeature-provider-tck/tests/conftest.py b/tools/openfeature-provider-tck/tests/conftest.py new file mode 100644 index 000000000..3f70730f0 --- /dev/null +++ b/tools/openfeature-provider-tck/tests/conftest.py @@ -0,0 +1,37 @@ +"""Known deviations of the Python SDK, recorded rather than hidden. + +A conformance suite that quietly goes green on scenarios it did not run is worse +than no suite at all -- and the same is true of one that quietly goes green on a +scenario it *did* run and fail. So the one scenario the Python SDK cannot +currently satisfy is marked ``xfail(strict=True)`` here, which: + +* keeps it visible in the report, as XFAIL with the reason attached; +* fails the suite if it ever *passes*, so the marker is removed the moment the + SDK is fixed rather than lingering as a lie. + +This lives in the TCK's own self-test rather than in the shared package. It is a +fact about the SDK under test, not part of the conformance definition, and +Appendix F deliberately leaves a general "known deviations" concept as an open +question (spec#417, Q4). If that concept lands, this moves into it. +""" + +from __future__ import annotations + +import pytest + +# The Scenario Outline row that asks for boolean-flag as an Integer. +_BOOL_AS_INT = "test_requesting_the_wrong_type_returns_the_code_default[boolean-flag-Integer-1]" + +_REASON = ( + "python-sdk: a boolean satisfies an Integer request. The client type-checks with " + "isinstance(value, int) and bool is a subclass of int in Python, so boolean-flag " + "requested as an Integer returns True with reason STATIC and no error code, where " + "the specification requires the code default and TYPE_MISMATCH. " + "See https://github.com/open-feature/python-sdk/issues/619" +) + + +def pytest_collection_modifyitems(items: list[pytest.Item]) -> None: + for item in items: + if item.name == _BOOL_AS_INT: + item.add_marker(pytest.mark.xfail(reason=_REASON, strict=True)) diff --git a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py new file mode 100644 index 000000000..3ebd240ec --- /dev/null +++ b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py @@ -0,0 +1,51 @@ +"""Run the conformance suite against the TCK's own updatable in-memory provider. + +This is the suite that exercises the configuration-change path, and it exists +because the SDK's in-memory provider cannot: it has no way to update a flag set, +so ``test_in_memory_conformance`` necessarily skips those scenarios. Without +this suite the change-event step definitions would ship with no coverage at all, +and a break in them would first surface in a containerised provider suite where +it looks like a provider defect. + +It is also the reference for what an in-process control path looks like when the +provider does support updates, which is what a file-based or +environment-variable provider should be able to do. +""" + +from __future__ import annotations + +import pytest +from pytest_bdd import scenarios + +from openfeature.contrib.tools.provider_tck import ( + Capability, + InProcessControl, + TckConfig, + features_path, +) + + +@pytest.fixture(scope="session") +def tck_config() -> TckConfig: + """Declare the provider under test and what it can do. + + ``STALE`` and ``UNAVAILABLE_INIT`` stay undeclared: there is still no + connection to lose, and ``InProcessControl`` does not implement + ``ConnectionControl``. ``CONFIGURATION_CHANGE`` is what this suite adds over + the plain in-memory one, and it is the whole point of it. + """ + control = InProcessControl() + return TckConfig( + name="controllable-in-memory", + control=control, + new_provider=control.new_provider, + capabilities={ + Capability.EVENTS, + Capability.CONFIGURATION_CHANGE, + Capability.OBJECT, + Capability.STRICT_NUMERIC_TYPING, + }, + ) + + +scenarios(features_path()) diff --git a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py new file mode 100644 index 000000000..de0250223 --- /dev/null +++ b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py @@ -0,0 +1,101 @@ +"""Run the conformance suite against the SDK's own in-memory provider. + +This is the TCK's self-test, and it earns its keep twice over. + +It is the **reference adoption** for a provider with no backend. Everything a +file-based or environment-variable provider has to write is here: one fixture +and one call. + +It is also the **Docker-free canary**. Needing no container and no network, it +runs in a fraction of a second, which makes it the fast check that catches a +broken step definition, a mis-wired capability gate or a regression in the +shared harness long before a containerised suite would. + +What it does not do is license providers that have a backend to test themselves +this way -- see ``BackendControl`` for why. +""" + +from __future__ import annotations + +import pytest +from pytest_bdd import scenarios + +from openfeature.contrib.tools.provider_tck import ( + Capability, + TckConfig, + canonical_flag_set, + features_path, +) +from openfeature.provider import FeatureProvider +from openfeature.provider.in_memory_provider import InMemoryProvider + + +class PlainMemoryControl: + """Backend control for the SDK's stock in-memory provider. + + ``prepare_scenario`` is a no-op because the provider is rebuilt from the + canonical flag set for every scenario, so each one already starts from an + untouched baseline. + + ``change_flag`` cannot be implemented at all, and the error says why. + Appendix A of the specification requires an SDK's in-memory provider to + "support a means of updating the flag set, resulting in the emission of + PROVIDER_CONFIGURATION_CHANGED events"; the Python SDK's copies its mapping + in the constructor and exposes no way to change it. The suite below + therefore leaves ``CONFIGURATION_CHANGE`` undeclared and the scenario is + reported as skipped with its reason, which is the honest outcome. Reaching + this error would mean the capability had been declared anyway. + """ + + @property + def description(self) -> str: + return "the Python SDK's InMemoryProvider, rebuilt per scenario" + + def prepare_scenario(self) -> None: + return None + + def change_flag(self) -> None: + msg = ( + "openfeature.provider.in_memory_provider.InMemoryProvider cannot change its " + "flag set: it copies the mapping in its constructor and exposes no update " + "method, so a configuration change can be neither applied nor signalled. " + "Appendix A of the specification requires it. See " + "ControllableInMemoryProvider for what the SDK's provider is missing" + ) + raise NotImplementedError(msg) + + +def _new_provider() -> FeatureProvider: + return InMemoryProvider(canonical_flag_set()) + + +@pytest.fixture(scope="session") +def tck_config() -> TckConfig: + """Declare the provider under test and what it can do. + + Each omission is a fact about the provider rather than a convenience: + + * ``CONFIGURATION_CHANGE`` -- omitted because the SDK's in-memory provider + cannot update its flag set. That is a finding, not a configuration choice; + see ``PlainMemoryControl``. + * ``STALE`` and ``UNAVAILABLE_INIT`` -- omitted because there is no + connection to lose. ``PlainMemoryControl`` does not implement + ``ConnectionControl`` for the same reason, and the two omissions keep each + other honest: the scenarios are skipped before any step can reach an + operation the control cannot perform. + * ``TARGETING`` and ``CACHING`` -- omitted because no scenario carries their + tags yet, so leaving them out skips nothing. + """ + return TckConfig( + name="in-memory", + control=PlainMemoryControl(), + new_provider=_new_provider, + capabilities={ + Capability.EVENTS, + Capability.OBJECT, + Capability.STRICT_NUMERIC_TYPING, + }, + ) + + +scenarios(features_path()) diff --git a/tools/openfeature-provider-tck/tests/test_in_process_control.py b/tools/openfeature-provider-tck/tests/test_in_process_control.py new file mode 100644 index 000000000..7a100a2ce --- /dev/null +++ b/tools/openfeature-provider-tck/tests/test_in_process_control.py @@ -0,0 +1,139 @@ +"""Things the Gherkin cannot assert about itself. + +Each of these is a way the in-process control path could look correct while +quietly making the conformance suites meaningless. +""" + +from __future__ import annotations + +import pytest + +from openfeature.contrib.tools.provider_tck import ( + CHANGING_FLAG_KEY, + ConnectionControl, + ControllableInMemoryProvider, + InProcessControl, + canonical_flag_set, +) +from openfeature.event import ProviderEvent + + +def _resolve_changing(provider: ControllableInMemoryProvider) -> str: + return provider.resolve_string_details(CHANGING_FLAG_KEY, "unset").value + + +def test_change_flag_actually_changes_the_resolved_value() -> None: + """The assumption every configuration-change scenario rests on. + + If ``change_flag`` emitted an event without altering what the provider + resolves, the scenario would still pass its event assertion and the suite + would be certifying a signal with nothing behind it. + """ + control = InProcessControl() + provider = control.new_provider() + assert isinstance(provider, ControllableInMemoryProvider) + + before = _resolve_changing(provider) + control.change_flag() + after = _resolve_changing(provider) + + assert before != after, "change_flag did not change the resolved value" + + +def test_change_flag_emits_a_configuration_change_event_naming_the_flag() -> None: + """The event the scenarios await is the provider's own, and it names the flag.""" + control = InProcessControl() + provider = control.new_provider() + + seen: list[tuple[ProviderEvent, list[str] | None]] = [] + + def record(_provider: object, event: ProviderEvent, details: object) -> None: + seen.append((event, getattr(details, "flags_changed", None))) + + # attach() is how the SDK registry wires a provider's emitter; doing it by + # hand keeps this a unit test of the provider rather than of the registry. + provider.attach(record) + control.change_flag() + + assert seen, "no event was emitted" + event, flags_changed = seen[-1] + assert event is ProviderEvent.PROVIDER_CONFIGURATION_CHANGED + assert flags_changed == [CHANGING_FLAG_KEY] + + +def test_change_does_not_leak_into_the_next_scenario() -> None: + """Scenario isolation. + + A leak here would make the suite order-dependent: a scenario running after + the configuration-change one would start with ``changing-flag`` already + flipped, and the failure would look like a provider defect. + """ + control = InProcessControl() + + first = control.new_provider() + assert isinstance(first, ControllableInMemoryProvider) + baseline = _resolve_changing(first) + + control.change_flag() + assert _resolve_changing(first) != baseline, "precondition: change_flag had no effect" + + control.prepare_scenario() + + second = control.new_provider() + assert isinstance(second, ControllableInMemoryProvider) + assert _resolve_changing(second) == baseline, ( + "the next scenario did not start from the baseline" + ) + + +def test_change_flag_without_a_provider_fails_clearly() -> None: + """In-process the flag store and the provider are the same object, so there is + nothing to change before one exists. Saying so beats an AttributeError.""" + control = InProcessControl() + with pytest.raises(RuntimeError, match="must create one"): + control.change_flag() + + +def test_in_process_control_does_not_pretend_to_have_a_connection() -> None: + """The load-bearing one. + + A no-op ``disconnect`` would report the ``@stale`` scenarios as passed + against a provider that cannot go stale -- precisely the silent-green + failure a conformance suite must never have. ``InProcessControl`` therefore + does not implement ``ConnectionControl`` at all, and the TCK turns that into + a skip with a reason. + """ + assert not isinstance(InProcessControl(), ConnectionControl), ( + "InProcessControl implements ConnectionControl: an in-memory provider has no " + "connection to lose, and a no-op implementation would make the @stale " + "scenarios pass without testing anything" + ) + + +def test_canonical_flag_set_omits_missing_flag() -> None: + """The property the FLAG_NOT_FOUND scenario depends on. + + Seeding ``missing-flag`` would turn that scenario green for the wrong + reason, and nothing else in the suite would notice. + """ + assert "missing-flag" not in canonical_flag_set() + + +def test_update_flags_names_the_union_of_old_and_new_keys() -> None: + """Appendix A asks for the union, not just the new keys. + + A consumer caching evaluations needs to know everything that might have + changed, and a key that disappeared has changed as much as one that arrived. + """ + provider = ControllableInMemoryProvider(canonical_flag_set()) + + seen: list[list[str] | None] = [] + provider.attach(lambda _p, _e, details: seen.append(details.flags_changed)) + + provider.update_flags({}) + + assert seen, "no event was emitted" + assert seen[-1] is not None + assert set(seen[-1]) == set(canonical_flag_set()), ( + "the event did not name every flag that disappeared" + ) From 7f281943402ef0644e280d1defe291e6c20046ea Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 13:57:43 +0200 Subject: [PATCH 02/46] fix(provider-tck): apply ruff format, and run the package in CI Two things CI caught that local verification did not. `ruff format` is a separate pre-commit hook from `ruff check`, and only the latter was run locally. Nine files needed reformatting; the changes are cosmetic line-wrapping only. More importantly, the package was not being tested in CI at all. The build matrix is gated on dorny/paths-filter and its filter list had no entry for tools/openfeature-provider-tck, so no change under that path expanded the matrix and the suite never ran. The locally reported 56 passed / 7 skipped / 2 xfailed was local-only. Adding the filter block, mirroring the one for tools/openfeature-flagd-core, turns it on. Verified after formatting: 56 passed, 7 skipped, 2 xfailed; ruff check and mypy --strict still clean. Signed-off-by: Simon Schrottner --- .github/workflows/build.yml | 3 +++ .../openfeature/contrib/tools/provider_tck/config.py | 9 +++++++-- .../openfeature/contrib/tools/provider_tck/control.py | 4 +++- .../openfeature/contrib/tools/provider_tck/plugin.py | 1 + .../openfeature/contrib/tools/provider_tck/provider.py | 5 ++++- .../contrib/tools/provider_tck/steps/event_steps.py | 4 +--- .../contrib/tools/provider_tck/steps/flag_steps.py | 10 ++++++---- .../openfeature/contrib/tools/provider_tck/values.py | 6 +++++- tools/openfeature-provider-tck/tests/conftest.py | 4 +++- .../tests/test_in_process_control.py | 4 +++- 10 files changed, 36 insertions(+), 14 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 8c09e5148..d80adb191 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -66,6 +66,9 @@ jobs: tools/openfeature-flagd-api-testkit: - 'tools/openfeature-flagd-api-testkit/**' - 'uv.lock' + tools/openfeature-provider-tck: + - 'tools/openfeature-provider-tck/**' + - 'uv.lock' build: needs: changes diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py index 468a4921c..b3e01fab7 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py @@ -118,7 +118,9 @@ def __post_init__(self) -> None: "fits your provider" ) if self.new_provider is None: - problems.append("new_provider is required: the TCK has nothing to test without it") + problems.append( + "new_provider is required: the TCK has nothing to test without it" + ) # Normalise whatever iterable the caller passed into a frozenset, so a # set literal, a list or a generator all behave the same. @@ -131,7 +133,10 @@ def __post_init__(self) -> None: f"the Capability enum" ) - if Capability.UNAVAILABLE_INIT in self.capabilities and self.new_unavailable_provider is None: + if ( + Capability.UNAVAILABLE_INIT in self.capabilities + and self.new_unavailable_provider is None + ): problems.append( "capabilities declares Capability.UNAVAILABLE_INIT but " "new_unavailable_provider is None: the @unavailable scenarios need a " diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py index bfa3064be..0e83e5bd2 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py @@ -96,7 +96,9 @@ def reconnect(self) -> None: """ -def unsupported_control(control: BackendControl, operation: str) -> UnsupportedControlError: +def unsupported_control( + control: BackendControl, operation: str +) -> UnsupportedControlError: """Build the error raised when a backend has no connection to control. The message names the fix, because the mistake it reports is always the same diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py index 239c41acd..b8b1a73b1 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py @@ -30,6 +30,7 @@ "openfeature.contrib.tools.provider_tck.steps.event_steps", ] + def pytest_configure(config: pytest.Config) -> None: """Register the capability tags as markers. diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py index 5b1c9faa4..33de67762 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py @@ -83,7 +83,10 @@ def flag(self, key: str) -> InMemoryFlag[typing.Any] | None: def changing_flag(default_variant: str) -> InMemoryFlag[str]: return InMemoryFlag( default_variant=default_variant, - variants={_CHANGING_BASELINE: _CHANGING_BASELINE, _CHANGING_CHANGED: _CHANGING_CHANGED}, + variants={ + _CHANGING_BASELINE: _CHANGING_BASELINE, + _CHANGING_CHANGED: _CHANGING_CHANGED, + }, ) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py index 47a37da8f..6b6356995 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py @@ -128,9 +128,7 @@ def the_flag_should_be_part_of_the_event_payload(tck_state: TckState) -> None: f"name {key!r}" ) else: - msg = ( - f"the configuration-change event named {changed}, expected it to include {key!r}" - ) + msg = f"the configuration-change event named {changed}, expected it to include {key!r}" raise AssertionError(msg) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py index f45b8dbff..f284eb5b9 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py @@ -88,7 +88,11 @@ def the_resolved_value_should_be(tck_state: TckState, expected: str) -> None: wanted = parse_value(flag_type, expected) if not values_equal(wanted, record.value): - detail = f" (the client also reported: {record.error_message})" if record.error_message else "" + detail = ( + f" (the client also reported: {record.error_message})" + if record.error_message + else "" + ) msg = ( f"flag {tck_state.flag_key!r} resolved to {describe(record.value)}, " f"expected {describe(wanted)}{detail}" @@ -187,9 +191,7 @@ def the_resolved_object_value_should_contain( raise AssertionError(msg) actual = record.value[key] if not values_equal(wanted, actual): - msg = ( - f"object member {key!r} was {describe(actual)}, expected {describe(wanted)}" - ) + msg = f"object member {key!r} was {describe(actual)}, expected {describe(wanted)}" raise AssertionError(msg) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py index 13dbe696e..4459d4660 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py @@ -92,7 +92,11 @@ def values_equal(expected: typing.Any, actual: typing.Any) -> bool: # rule would make True == 1 and quietly satisfy the scenario that exists to # catch exactly that confusion. if isinstance(expected, bool) or isinstance(actual, bool): - return isinstance(expected, bool) and isinstance(actual, bool) and expected == actual + return ( + isinstance(expected, bool) + and isinstance(actual, bool) + and expected == actual + ) expected_number = _as_number(expected) if expected_number is not None: diff --git a/tools/openfeature-provider-tck/tests/conftest.py b/tools/openfeature-provider-tck/tests/conftest.py index 3f70730f0..a5e6726f5 100644 --- a/tools/openfeature-provider-tck/tests/conftest.py +++ b/tools/openfeature-provider-tck/tests/conftest.py @@ -20,7 +20,9 @@ import pytest # The Scenario Outline row that asks for boolean-flag as an Integer. -_BOOL_AS_INT = "test_requesting_the_wrong_type_returns_the_code_default[boolean-flag-Integer-1]" +_BOOL_AS_INT = ( + "test_requesting_the_wrong_type_returns_the_code_default[boolean-flag-Integer-1]" +) _REASON = ( "python-sdk: a boolean satisfies an Integer request. The client type-checks with " diff --git a/tools/openfeature-provider-tck/tests/test_in_process_control.py b/tools/openfeature-provider-tck/tests/test_in_process_control.py index 7a100a2ce..88322acfd 100644 --- a/tools/openfeature-provider-tck/tests/test_in_process_control.py +++ b/tools/openfeature-provider-tck/tests/test_in_process_control.py @@ -75,7 +75,9 @@ def test_change_does_not_leak_into_the_next_scenario() -> None: baseline = _resolve_changing(first) control.change_flag() - assert _resolve_changing(first) != baseline, "precondition: change_flag had no effect" + assert _resolve_changing(first) != baseline, ( + "precondition: change_flag had no effect" + ) control.prepare_scenario() From 2ffd333d95fa93258b845a9af9cff7962b6997a0 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 14:01:43 +0200 Subject: [PATCH 03/46] fix(provider-tck): add the package to uv.lock `uv sync --frozen` in the build workflow validates the lockfile against the manifests, and the previous commit added openfeature-provider-tck to the workspace root's dependencies and [tool.uv.sources] without regenerating the lock. That breaks the build job for *every* package, not just this one. It was latent until now only because the paths-filter had no entry for this package, so no build job ran at all. Enabling the filter in the previous commit would have surfaced it as a red build. The regeneration also picks up openfeature-provider-flagd 0.5.1 -> 0.5.2, which the lock had missed when that release landed. Signed-off-by: Simon Schrottner --- uv.lock | 38 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/uv.lock b/uv.lock index 3168e2514..b0ab379c1 100644 --- a/uv.lock +++ b/uv.lock @@ -17,6 +17,7 @@ members = [ "openfeature-provider-flagd", "openfeature-provider-flipt", "openfeature-provider-ofrep", + "openfeature-provider-tck", "openfeature-provider-unleash", "openfeature-python-contrib", ] @@ -843,7 +844,7 @@ name = "exceptiongroup" version = "1.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", marker = "python_full_version < '3.13'" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/50/79/66800aadf48771f6b62f7eb014e352e5d06856655206165d775e675a02c9/exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219", size = 30371, upload-time = "2025-11-21T23:01:54.787Z" } wheels = [ @@ -1868,7 +1869,7 @@ dev = [ [[package]] name = "openfeature-provider-flagd" -version = "0.5.1" +version = "0.5.2" source = { editable = "providers/openfeature-provider-flagd" } dependencies = [ { name = "cachebox" }, @@ -1989,6 +1990,37 @@ dev = [ { name = "types-requests", specifier = ">=2.32.0,<3.0.0" }, ] +[[package]] +name = "openfeature-provider-tck" +version = "0.1.0" +source = { editable = "tools/openfeature-provider-tck" } +dependencies = [ + { name = "openfeature-sdk" }, + { name = "pytest" }, + { name = "pytest-bdd" }, +] + +[package.dev-dependencies] +dev = [ + { name = "coverage", extra = ["toml"] }, + { name = "mypy" }, + { name = "poethepoet" }, +] + +[package.metadata] +requires-dist = [ + { name = "openfeature-sdk", specifier = ">=0.8.2" }, + { name = "pytest", specifier = ">=8.4.0" }, + { name = "pytest-bdd", specifier = ">=8.1.0,<9.0.0" }, +] + +[package.metadata.requires-dev] +dev = [ + { name = "coverage", extras = ["toml"], specifier = ">=7.10.0,<8.0.0" }, + { name = "mypy", specifier = ">=1.18.0,<2.0.0" }, + { name = "poethepoet", specifier = ">=0.37.0" }, +] + [[package]] name = "openfeature-provider-unleash" version = "0.1.2" @@ -2042,6 +2074,7 @@ dependencies = [ { name = "openfeature-provider-flagd" }, { name = "openfeature-provider-flipt" }, { name = "openfeature-provider-ofrep" }, + { name = "openfeature-provider-tck" }, { name = "openfeature-provider-unleash" }, ] @@ -2063,6 +2096,7 @@ requires-dist = [ { name = "openfeature-provider-flagd", editable = "providers/openfeature-provider-flagd" }, { name = "openfeature-provider-flipt", editable = "providers/openfeature-provider-flipt" }, { name = "openfeature-provider-ofrep", editable = "providers/openfeature-provider-ofrep" }, + { name = "openfeature-provider-tck", editable = "tools/openfeature-provider-tck" }, { name = "openfeature-provider-unleash", editable = "providers/openfeature-provider-unleash" }, ] From 8324d04b3472e8e43a0dd26424186ab2d1e0aacc Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 14:02:54 +0200 Subject: [PATCH 04/46] fix(provider-tck): make ready_timeout actually bound initialisation TckConfig.ready_timeout was documented but never read by anything, so a provider that hung while connecting would hang the whole pytest session with no useful message, and the documented knob did nothing. api.set_provider initialises synchronously and has no timeout of its own, so the bound comes from running it on a worker thread and giving up on the result. The worker is deliberately not cancelled -- Python cannot interrupt a thread blocked in a socket call -- and is left to finish or die with the process, which is acceptable because a timeout already means the scenario is failing. A config field that claims to do something it does not is exactly the kind of quiet untruth this suite exists to catch, so it is fixed rather than removed. Verified: 56 passed, 7 skipped, 2 xfailed; ruff and mypy --strict clean. Signed-off-by: Simon Schrottner --- .../provider_tck/steps/provider_steps.py | 35 ++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py index 057b24842..bca37faee 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py @@ -2,11 +2,13 @@ from __future__ import annotations +import concurrent.futures import contextlib from pytest_bdd import given, parsers from openfeature import api +from openfeature.provider import FeatureProvider from ..state import TckState @@ -30,7 +32,14 @@ def a_stable_provider(tck_state: TckState) -> None: raise AssertionError(msg) try: - api.set_provider(provider, config.domain) + _set_provider_within(provider, config.domain, config.ready_timeout) + except TimeoutError: + msg = ( + f"the provider did not become ready within {config.ready_timeout}s. The backend " + f"is up and seeded at this point, so either initialisation is genuinely hanging " + f"or TckConfig.ready_timeout is too short" + ) + raise AssertionError(msg) from None except Exception as exc: msg = ( f"registering the provider raised {exc!r}. The backend is up and seeded " @@ -79,3 +88,27 @@ def an_unavailable_provider(tck_state: TckState) -> None: api.set_provider(provider, config.domain) tck_state.client = api.get_client(config.domain) + + +def _set_provider_within( + provider: FeatureProvider, domain: str, timeout: float +) -> None: + """Register a provider, giving up if initialisation has not returned in time. + + ``api.set_provider`` initialises synchronously and has no timeout of its own, so a + provider that hangs while connecting would hang the whole session with no useful + message. Running it on a worker thread bounds it. + + The worker is deliberately not cancelled on timeout -- Python cannot interrupt a + thread blocked in a socket call -- so it is left to finish or die with the process. + That is acceptable here because a timeout already means the scenario is failing. + """ + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: + future = pool.submit(api.set_provider, provider, domain) + try: + future.result(timeout=timeout) + except concurrent.futures.TimeoutError: + raise TimeoutError from None + finally: + # Do not block __exit__ on a worker that is still stuck. + pool.shutdown(wait=False) From 15a57bb6ec4cbaeace1db99e1dab212cea8dbf9e Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 14:12:02 +0200 Subject: [PATCH 05/46] fix(provider-tck): accept any capability collection, and type-check the tests TckConfig.capabilities was annotated frozenset[Capability], but the README tells adopters to write `capabilities={Capability.EVENTS, ...}` -- a set literal. Anyone copying the documented example and running mypy got an incompatible-argument error from the suite's own documentation. It is now annotated Collection[Capability], which is what __post_init__ already accepted: a set, a list or a generator all normalise to a frozenset on construction. The reason this was invisible is the second half of the fix. mypy was configured `files = "src"`, so the tests were never checked -- and the tests are the reference adoption, the thing an adopting provider copies. They are now in scope, which is what would have caught the annotation in the first place. Verified: mypy clean over src and tests (17 files), ruff format and check clean, 56 passed / 7 skipped / 2 xfailed. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/pyproject.toml | 2 +- .../src/openfeature/contrib/tools/provider_tck/config.py | 9 +++++++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/tools/openfeature-provider-tck/pyproject.toml b/tools/openfeature-provider-tck/pyproject.toml index cdddc736c..3679c440e 100644 --- a/tools/openfeature-provider-tck/pyproject.toml +++ b/tools/openfeature-provider-tck/pyproject.toml @@ -47,7 +47,7 @@ packages = ["src/openfeature"] [tool.mypy] mypy_path = "src" -files = "src" +files = ["src", "tests"] python_version = "3.10" namespace_packages = true explicit_package_bases = true diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py index b3e01fab7..77b783cdb 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py @@ -2,7 +2,7 @@ from __future__ import annotations -from collections.abc import Callable, Iterable +from collections.abc import Callable, Collection, Iterable from dataclasses import dataclass, field from openfeature.provider import FeatureProvider @@ -79,9 +79,14 @@ class TckConfig: skipped with the reason reported. """ - capabilities: frozenset[Capability] = field(default=ALL_CAPABILITIES) + capabilities: Collection[Capability] = field(default=ALL_CAPABILITIES) """Which optional parts of the provider contract this provider supports. + Typed as a ``Collection`` rather than a ``frozenset`` so that the obvious + thing to write -- a set literal, which is what the README shows -- is also + the correctly typed thing to write. It is normalised to a frozenset on + construction, so a list, a set or a generator all behave identically. + Scenarios tagged with an undeclared capability are reported as skipped with the reason, never as passed. Defaults to everything; narrow it rather than widening it. From 8df3b7c97fc8c0f32b6173df925fe30d287beb35 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 14:38:51 +0200 Subject: [PATCH 06/46] feat(provider-tck): source the conformance assets from the spec submodule The feature files, the canonical flag set and the control-API document are owned by open-feature/spec, not by this repository. Committing copies of them here forks the definition of conformance -- the one thing this suite exists to prevent -- and leaves no machine-checkable record of which spec revision the copies came from. Replace them with a git submodule at tools/openfeature-provider-tck/spec, pinned at dfa16586 (spec#423), plus a build-time copy. The copies are gitignored and carry a DO-NOT-EDIT marker, so the pin is now the only record of the revision and the two cannot drift apart unnoticed. An adopter installing this package still needs no submodule: the copies are force-included into the wheel and the sdist, and the sdist excludes the submodule itself so it carries the four assets rather than the whole spec repository. Only a contributor to this package needs the submodule, and `poe test` syncs it first. This mirrors what openfeature-flagd-api-testkit already does for the flagd test harness. Signed-off-by: Simon Schrottner --- .gitmodules | 3 + pyproject.toml | 5 +- tools/openfeature-provider-tck/.gitignore | 7 + tools/openfeature-provider-tck/README.md | 34 +- tools/openfeature-provider-tck/hatch_build.py | 51 +++ .../hatch_build_sync.py | 56 +++ tools/openfeature-provider-tck/pyproject.toml | 22 +- tools/openfeature-provider-tck/spec | 1 + .../contrib/tools/provider_tck/__init__.py | 23 +- .../tools/provider_tck/control-api.yaml | 368 ------------------ .../provider_tck/features/errors.feature | 80 ---- .../provider_tck/features/evaluation.feature | 59 --- .../provider_tck/features/events.feature | 42 -- .../provider_tck/features/lifecycle.feature | 33 -- .../flag_data/canonical-flags.json | 82 ---- 15 files changed, 187 insertions(+), 679 deletions(-) create mode 100644 tools/openfeature-provider-tck/.gitignore create mode 100644 tools/openfeature-provider-tck/hatch_build.py create mode 100644 tools/openfeature-provider-tck/hatch_build_sync.py create mode 160000 tools/openfeature-provider-tck/spec delete mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control-api.yaml delete mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/errors.feature delete mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/evaluation.feature delete mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/events.feature delete mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/lifecycle.feature delete mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/flag_data/canonical-flags.json diff --git a/.gitmodules b/.gitmodules index 7e8bf9ed5..31678c42e 100644 --- a/.gitmodules +++ b/.gitmodules @@ -7,3 +7,6 @@ [submodule "providers/openfeature-provider-flagd/openfeature/test-harness"] path = providers/openfeature-provider-flagd/openfeature/test-harness url = https://github.com/open-feature/flagd-testbed.git +[submodule "tools/openfeature-provider-tck/spec"] + path = tools/openfeature-provider-tck/spec + url = https://github.com/open-feature/spec diff --git a/pyproject.toml b/pyproject.toml index c1f1ce6bb..e250a4b79 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -60,7 +60,10 @@ exclude = [ ".venv", "__pycache__", "venv", - "providers/openfeature-provider-flagd/src/openfeature/schemas/**" + "providers/openfeature-provider-flagd/src/openfeature/schemas/**", + # Submodules of other repositories: not ours to lint or format. + "providers/openfeature-provider-flagd/openfeature/spec/**", + "tools/openfeature-provider-tck/spec/**", ] [tool.ruff.lint] diff --git a/tools/openfeature-provider-tck/.gitignore b/tools/openfeature-provider-tck/.gitignore new file mode 100644 index 000000000..066646223 --- /dev/null +++ b/tools/openfeature-provider-tck/.gitignore @@ -0,0 +1,7 @@ +# Copied from the open-feature/spec submodule by hatch_build_sync.py. +# DO NOT EDIT the copies, and do not commit them: the canonical definitions live +# in spec/specification/assets/provider-tck/, and the revision this package is +# built against is recorded by the submodule pin. +src/openfeature/contrib/tools/provider_tck/features/ +src/openfeature/contrib/tools/provider_tck/flag_data/ +src/openfeature/contrib/tools/provider_tck/control-api.yaml diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index d735a5c5e..af37eda19 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -60,7 +60,8 @@ writing test infrastructure, that is a defect here rather than something for you pytest-bdd generates one test per scenario — and one per row of a Scenario Outline — so failures name a scenario and `-k` selects one as usual. The feature files and canonical flag set are packaged -with the distribution, so **you need no git submodule**. +inside the distribution, so **adopting this package needs no git submodule** — see +[Where the assets come from](#where-the-assets-come-from). ### Timings @@ -168,6 +169,33 @@ Only half the machinery is missing — `AbstractProvider` already supplies `emit_provider_configuration_changed` — which is why `ControllableInMemoryProvider` here is a small subclass rather than a reimplementation, and why it should port back to the SDK as a method. +## Where the assets come from + +The Gherkin feature files, the canonical flag set and the control-API document are **not owned by +this repository**. They are the language-agnostic conformance artifacts defined in +[open-feature/spec][spec] under `specification/assets/provider-tck/`, and every language's TCK ships +the same ones — which is the only reason a conformance claim means the same thing in Python as it +does in Java. + +**Adopting this package needs no submodule.** The assets are copied into the wheel and the sdist at +build time, so `pip install openfeature-provider-tck` gives you everything the suite runs on. + +**Contributing to this package does.** The spec is a git submodule at +`tools/openfeature-provider-tck/spec`, and the copies under +`src/openfeature/contrib/tools/provider_tck/` are gitignored and generated: + +```bash +git submodule update --init tools/openfeature-provider-tck/spec +poe test # runs `poe sync-spec-assets` first +``` + +The copies carry a `DO-NOT-EDIT.txt` because editing them forks the definition of conformance, which +is the one thing this suite exists to prevent. A change goes to [open-feature/spec][spec] first; +then bump the submodule pin here. Committing no copies means the spec revision this package targets +is recorded by the pin and nowhere else, so the two cannot drift apart unnoticed. + +This mirrors what `openfeature-flagd-api-testkit` already does for the flagd test harness. + ## The self-tests | Suite | Subject | Why | @@ -184,10 +212,6 @@ No Docker, no network, under a second. ## Known gaps -- **The assets are vendored, not submoduled.** `features/` and `flag_data/` are copies of - `specification/assets/provider-tck/` in [open-feature/spec][spec]. Changes belong there and are - copied here; a follow-up will source them from a submodule at build time, as - `openfeature-flagd-api-testkit` already does for the flagd test harness. - **Evaluation context passthrough is unverifiable.** The scenarios build evaluation contexts but cannot assert one *reached* the backend. That needs an echo operation on the control API. - **No HTTP control client yet.** It arrives with the first containerised adopter. diff --git a/tools/openfeature-provider-tck/hatch_build.py b/tools/openfeature-provider-tck/hatch_build.py new file mode 100644 index 000000000..4b6f1d840 --- /dev/null +++ b/tools/openfeature-provider-tck/hatch_build.py @@ -0,0 +1,51 @@ +"""Hatch build hook to copy the canonical conformance assets into the package. + +The feature files, the canonical flag set and the control-API document are owned +by open-feature/spec and reach this package through a git submodule, so nothing +in this repository can fork the definition of conformance. They are copied into +the source tree at build time and force-included into the distribution, which is +what lets an *adopter* install the wheel and run the suite with no submodule of +their own. +""" + +import sys +from pathlib import Path + +from hatchling.builders.hooks.plugin.interface import BuildHookInterface + +# Hatchling loads this file by path rather than importing it as part of a +# package, so its directory is not on sys.path and the sibling sync module -- +# the single definition of what gets copied where -- would not be importable. +sys.path.insert(0, str(Path(__file__).parent)) + +from hatch_build_sync import FILES, PACKAGE_REL, SPEC_ASSETS, TREES, sync + + +class SpecAssetsCopyHook(BuildHookInterface): + PLUGIN_NAME = "spec-assets-copy" + + def initialize(self, version: str, build_data: dict) -> None: + root = Path(self.root) + copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES] + + # Building from a checkout: refresh from the submodule, so what ships is + # always the revision the pin names. Building from an sdist: there is no + # submodule, but the copies are already in the tree. + if SPEC_ASSETS.exists(): + sync() + elif not all(path.exists() for path in copies): + missing = ", ".join(str(p) for p in copies if not p.exists()) + msg = ( + f"Conformance assets missing ({missing}) and the open-feature/spec " + f"submodule is not checked out at {SPEC_ASSETS}. Run " + "`git submodule update --init tools/openfeature-provider-tck/spec`." + ) + raise FileNotFoundError(msg) + + # Force-include the gitignored copies into both sdist and wheel. + force = build_data.setdefault("force_include", {}) + for path in copies: + for member in [path] if path.is_file() else path.rglob("*"): + if member.is_file(): + rel = str(member.relative_to(root)) + force[rel] = rel diff --git a/tools/openfeature-provider-tck/hatch_build_sync.py b/tools/openfeature-provider-tck/hatch_build_sync.py new file mode 100644 index 000000000..f31bc55b5 --- /dev/null +++ b/tools/openfeature-provider-tck/hatch_build_sync.py @@ -0,0 +1,56 @@ +"""Copy the canonical conformance assets from the spec submodule into the package. + +Used by `poe sync-spec-assets` for local development and CI testing. The hatch +build hook (hatch_build.py) handles inclusion in the wheel and sdist. + +The assets are owned by open-feature/spec, not by this repository. Copying them +in at build time -- rather than committing copies -- means the spec revision this +package was built against is recorded by the submodule pin and nowhere else, so +the two cannot drift apart unnoticed. An *adopter* installing the wheel still +needs no submodule: the copies are inside the distribution. +""" + +import shutil +from pathlib import Path + +ROOT = Path(__file__).parent +SPEC_ASSETS = (ROOT / "spec/specification/assets/provider-tck").resolve() +PACKAGE_REL = Path("src/openfeature/contrib/tools/provider_tck") +DEST_BASE = ROOT / PACKAGE_REL + +DO_NOT_EDIT = ( + "Generated by hatch_build_sync.py from the open-feature/spec submodule.\n" + "DO NOT EDIT. Changes belong in open-feature/spec under\n" + "specification/assets/provider-tck/, then bump the submodule pin.\n" +) + +# (source directory or file, destination) relative to SPEC_ASSETS / DEST_BASE. +TREES = [("gherkin", "features"), ("flags", "flag_data")] +FILES = [("openapi/control-api.yaml", "control-api.yaml")] + + +def sync() -> None: + if not SPEC_ASSETS.exists(): + msg = ( + f"Conformance assets not found at {SPEC_ASSETS}. " + "Make sure submodules are initialized: " + "`git submodule update --init tools/openfeature-provider-tck/spec`." + ) + raise FileNotFoundError(msg) + + for src_name, dest_name in TREES: + dest = DEST_BASE / dest_name + if dest.exists(): + shutil.rmtree(dest) + shutil.copytree(SPEC_ASSETS / src_name, dest) + (dest / "DO-NOT-EDIT.txt").write_text(DO_NOT_EDIT, encoding="utf-8") + + for src_name, dest_name in FILES: + dest = DEST_BASE / dest_name + if dest.exists(): + dest.unlink() + shutil.copy2(SPEC_ASSETS / src_name, dest) + + +if __name__ == "__main__": + sync() diff --git a/tools/openfeature-provider-tck/pyproject.toml b/tools/openfeature-provider-tck/pyproject.toml index 3679c440e..ff0cbe43a 100644 --- a/tools/openfeature-provider-tck/pyproject.toml +++ b/tools/openfeature-provider-tck/pyproject.toml @@ -42,8 +42,25 @@ dev = [ "poethepoet>=0.37.0", ] +[tool.hatch.build.targets.sdist] +# The conformance assets are gitignored copies of the spec submodule; the build +# hook force-includes them so an sdist builds into a wheel without a submodule. +force-include = {} +# Which is why the submodule itself has no business in the sdist: it is the whole +# spec repository, and only the copies of the four assets are needed downstream. +exclude = ["/spec"] + [tool.hatch.build.targets.wheel] packages = ["src/openfeature"] +# Ship the conformance assets even though they are gitignored: an adopter +# installing this package must need no submodule of their own. +artifacts = [ + "src/openfeature/contrib/tools/provider_tck/features/", + "src/openfeature/contrib/tools/provider_tck/flag_data/", + "src/openfeature/contrib/tools/provider_tck/control-api.yaml", +] + +[tool.hatch.build.hooks.custom] [tool.mypy] mypy_path = "src" @@ -62,8 +79,9 @@ disallow_any_generics = false omit = ["tests/**"] [tool.poe.tasks] -test = "pytest tests" -test-cov = "coverage run -m pytest tests" +sync-spec-assets = "python hatch_build_sync.py" +test = ["sync-spec-assets", {cmd = "pytest tests"}] +test-cov = ["sync-spec-assets", {cmd = "coverage run -m pytest tests"}] cov-report = "coverage xml" cov = ["test-cov", "cov-report"] mypy = "mypy" diff --git a/tools/openfeature-provider-tck/spec b/tools/openfeature-provider-tck/spec new file mode 160000 index 000000000..dfa16586d --- /dev/null +++ b/tools/openfeature-provider-tck/spec @@ -0,0 +1 @@ +Subproject commit dfa16586d91ca020ef1b3b82a7c972d833ff8f29 diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py index 31e9d39d3..8b615296d 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py @@ -81,13 +81,22 @@ def tck_config(): # NOTE ON THE SOURCE OF TRUTH # -# The files under features/ and flag_data/ are NOT owned by this repository. -# They are copies of the language-agnostic conformance artifacts defined in -# open-feature/spec under specification/assets/provider-tck/. They are vendored -# here so adopting this TCK never requires a git submodule of your own. Changes -# belong in open-feature/spec first and are copied here -- editing them locally -# forks the definition of conformance, which is the one thing this suite exists -# to prevent. See https://github.com/open-feature/spec/issues/417. +# The files under features/ and flag_data/, and control-api.yaml, are NOT owned +# by this repository and are NOT committed to it. They are copies of the +# language-agnostic conformance artifacts defined in open-feature/spec under +# specification/assets/provider-tck/, which reaches this package as a git +# submodule at tools/openfeature-provider-tck/spec and is copied in at build +# time by hatch_build.py. The copies are gitignored, so the only record of which +# spec revision this package targets is the submodule pin, and the two cannot +# drift apart unnoticed. +# +# They are copied into the distribution, so an adopter installing this package +# needs no submodule of their own; only a contributor to this package does. +# +# Changes belong in open-feature/spec first, followed by a bump of the submodule +# pin -- editing the copies locally forks the definition of conformance, which is +# the one thing this suite exists to prevent. +# See https://github.com/open-feature/spec/issues/417. _PACKAGE = "openfeature.contrib.tools.provider_tck" diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control-api.yaml b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control-api.yaml deleted file mode 100644 index fd9bc7000..000000000 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control-api.yaml +++ /dev/null @@ -1,368 +0,0 @@ -openapi: 3.0.3 - -info: - title: OpenFeature Provider TCK — Backend Control API - version: 0.0.1 - description: | - The control API that a **backend under test** must expose so the OpenFeature - Provider TCK can drive it. - - The TCK verifies the *provider contract*: how a provider maps backend - responses to typed resolution details, lifecycle states and events. To do - that it must be able to put the backend into specific states on demand — - running, unreachable, reconfigured. This document standardises how. - - This specification is derived from the control endpoints already implemented - by [`flagd-testbed`](https://github.com/open-feature/flagd-testbed)'s - "launchpad" server, which is the reference implementation. - - ## Where this document should live - - This file currently ships inside the Java `provider-tck` artifact, but it is - not a Java artifact: it is a language-agnostic contract that every language's - TCK must implement identically, and that backend vendors implement in - whatever language their testbed is written in (Go, for flagd). - - It therefore belongs in the OpenFeature **spec** repository - (`open-feature/spec`), alongside the canonical Gherkin feature files and the - canonical flag set. Those three artifacts are a single unit — a feature file - that evaluates `boolean-flag` is meaningless without the flag definition, and - a disconnect scenario is meaningless without the endpoint that produces the - disconnect. Splitting them across repositories would let them drift. - - Each language's TCK then vendors the spec repo (git submodule or equivalent) - and packages these files into its own distribution format, so that adopting a - TCK never requires a consumer to check out a submodule of their own. - - ## Conformance language - - The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT and MAY are to be - interpreted as described in RFC 2119. - - Each operation below is tagged **REQUIRED** or **OPTIONAL**. A backend that - implements every REQUIRED operation can run the full TCK. OPTIONAL operations - have a defined fallback that the TCK applies automatically, so omitting them - costs nothing but precision. - - --- - - ## Normative requirement 1 — the no-container-restart invariant - - > **Container lifecycle operations MUST NOT be used to simulate backend - > unavailability. Backend unavailability MUST be simulated from inside the - > running stack.** - - The TCK starts the vendor's Docker Compose stack **once per test suite** and - reads the dynamically mapped host ports. Testcontainers cannot reliably - preserve mapped ports across a container stop/start in all language - bindings — a restarted container generally comes back on a *different* host - port, which silently invalidates every provider instance already pointed at - the old one. Any TCK implementation in any language hits this, so the - constraint is part of the contract rather than a Java detail. - - Therefore an implementation of `/stop`, `/restart` or any other outage - simulation MUST achieve the outage by one of: - - * killing or suspending the backend **process** inside its container - (the reference behaviour — this is what flagd-testbed does); - * a proxy in the stack refusing or blackholing connections - (e.g. a toxiproxy toxic, an envoy `direct_response`); - * an in-container firewall or socket-level block. - - An implementation MUST NOT `docker stop`, `docker kill`, `docker rm` or - recreate any container in the stack while the suite is running. The stack is - brought up before the first scenario and torn down after the last one, and - the mapped ports MUST remain stable for that entire window. - - --- - - ## Normative requirement 2 — flag state semantics across outages - - Outage simulation and flag-state seeding are orthogonal, and the TCK relies - on that separation for scenario isolation: - - * `POST /start` **MUST** (re)seed flag state to the baseline defined by the - named configuration. Any mutation previously applied by `POST /change` - MUST be discarded. This is what makes `/start` usable as a reset. - * `POST /restart` and a `POST /stop` followed by a `POST /start` **of the - same configuration** MUST leave the backend serving the same baseline - flag state it served before the outage. An outage MUST NOT be observable - as a change in flag *values* — only as a change in *availability*. - * `POST /change` mutations persist until the next `/start` or `/reset`. - - --- - - ## Normative requirement 3 — compose stack conventions - - The backend under test is delivered as a **Docker Compose stack**, not a - single image, so vendors can compose proxies, edge services or several - containers. The TCK only relies on these conventions: - - * One service — by default named `backend`, overridable by the provider - author — exposes the control API on container-internal port `8080` - (also overridable). - * The same stack exposes whatever port(s) the provider connects to. - * **All external ports are dynamically mapped.** A stack MUST NOT pin host - ports; the TCK discovers them after startup and hands them to the - provider factory. - * The stack MAY contain any number of additional services. - - --- - - ## Known gap — evaluation context passthrough - - There is currently no operation for asserting that an evaluation context sent - by the provider actually reached the backend intact. Verifying that requires - an echo mechanism (e.g. `GET /last-evaluation` returning the most recent - request the backend received). Until such an operation exists, context - passthrough is out of scope for the TCK. - - license: - name: Apache 2.0 - url: https://www.apache.org/licenses/LICENSE-2.0 - -servers: - - url: http://{host}:{port} - description: | - Resolved at runtime from the Compose stack. `host` is the Docker host and - `port` is the dynamically mapped host port for the control service's - internal port 8080. - variables: - host: - default: localhost - port: - default: "8080" - -tags: - - name: lifecycle - description: Start and stop the backend process. - - name: availability - description: Simulate outages without touching containers. - - name: flags - description: Seed and mutate flag configuration. - - name: health - description: Readiness of the control API itself. - -paths: - - /start: - post: - tags: [lifecycle] - operationId: start - summary: "[REQUIRED] Start the backend and seed flags to a named baseline" - description: | - Starts the backend process using the named configuration and seeds flag - state to that configuration's baseline. - - MUST be idempotent in the sense that calling it while the backend is - already running is not an error: the implementation restarts the process - (or otherwise ensures it is running) with the requested configuration. - - Because this operation resets flag state, the TCK uses it as its default - scenario-isolation mechanism when `/reset` is not implemented. - - The set of valid configuration names is vendor-defined. Every - implementation MUST support the name `default`, which MUST serve the - canonical flag set the TCK's feature files assume. - - Reference implementation: flagd-testbed launches the `flagd` binary with - the config file of that name from `launchpad/configs` and rewrites - `/flags/allFlags.json`. - parameters: - - name: config - in: query - required: false - description: | - Name of the configuration to start with. Defaults to `default`. - schema: - type: string - default: default - example: default - responses: - "200": - description: Backend started and flag state seeded. - "400": - description: Unknown configuration name. - content: - application/json: - schema: - $ref: "#/components/schemas/Error" - - /stop: - post: - tags: [availability] - operationId: stop - summary: "[REQUIRED] Make the backend unreachable" - description: | - Makes the backend unreachable to the provider, simulating an outage. - - **MUST NOT stop the container.** See normative requirement 1. The - reference implementation kills the flagd process while its container - keeps running. - - The backend stays unreachable until a subsequent `POST /start`. Calling - `/stop` when the backend is already stopped MUST succeed. - - The TCK uses this to drive providers into `STALE` and `ERROR` states and - to assert `PROVIDER_STALE` / `PROVIDER_ERROR` events. - responses: - "200": - description: Backend is now unreachable; container still running. - - /restart: - post: - tags: [availability] - operationId: restart - summary: "[REQUIRED] Simulate an outage of a bounded duration" - description: | - Makes the backend unreachable, waits `seconds`, then starts it again with - the configuration currently in effect. - - Flag state MUST be preserved across the outage — see normative - requirement 2. This is what distinguishes `/restart` from - `/stop` + `/start`: the former is an availability event, the latter is - also a reset. - - This operation MAY return as soon as the outage has begun rather than - blocking for the full duration; the TCK does not rely on the response - being delayed. It awaits provider events instead. - - The TCK uses this for the disconnect/reconnect scenarios: `STALE` → - `PROVIDER_STALE`, then back to `READY` → `PROVIDER_READY`. - parameters: - - name: seconds - in: query - required: false - description: | - How long the backend stays unreachable. Defaults to 5. - - Providers differ enormously in how fast they notice an outage — - a streaming provider may see it in milliseconds while a polling - provider needs up to a full poll interval. Feature files therefore - parameterise this value and provider authors tune the matching - await timeouts. - schema: - type: integer - format: int32 - minimum: 0 - default: 5 - example: 5 - responses: - "200": - description: Outage started (and, for blocking implementations, ended). - - /change: - post: - tags: [flags] - operationId: change - summary: "[REQUIRED] Mutate flag configuration so the provider observes a change" - description: | - Mutates the flag configuration such that a conforming provider observes a - configuration change and, on re-evaluation, resolves a **different value** - for the affected flag. - - The implementation MUST: - - * change the resolved value of the flag with key `changing-flag`; - * do so without restarting the backend process, so that a provider sees - a configuration-change signal rather than a reconnect; - * make the change durable until the next `/start` or `/reset`. - - The implementation SHOULD toggle between exactly two known values so that - repeated calls are meaningful and the test remains deterministic - regardless of how many times it has run against the same stack. The - reference implementation toggles `changing-flag`'s `defaultVariant` - between `foo` and `bar`. - - The TCK uses this to assert `PROVIDER_CONFIGURATION_CHANGED`, that the - changed flag key appears in the event payload, and that a subsequent - evaluation returns the new value. - responses: - "200": - description: Flag configuration mutated. - - /reset: - post: - tags: [flags] - operationId: reset - summary: "[OPTIONAL] Restore the seeded baseline without an outage" - description: | - Restores flag state to the baseline of the configuration currently in - effect, discarding any mutation applied by `/change`, **without** making - the backend unreachable at any point. - - This is the preferred scenario-isolation primitive: unlike `/start` it - causes no availability blip, so it cannot inject spurious lifecycle - events into the next scenario. - - **Scope.** This operation resets flag state only. It MUST NOT be - expected to start a backend that is currently stopped — that is what - `/start` is for. A TCK therefore uses `/reset` only when the backend is - known to be running, and `/start` otherwise. The reference client tracks - this: `/stop` and `/restart` mark the backend as possibly-unreachable, so - the scenario that follows either of them is prepared with `/start`. - - **Fallback when not implemented.** A backend that does not implement this - operation MUST respond `404` or `501`. The TCK then falls back to - `POST /start?config={defaultConfig}`, which resets flag state at the cost - of a process restart. The fallback is detected once per suite and cached. - - Implementing `/reset` is RECOMMENDED for providers whose reconnect - behaviour makes the `/start` blip hard to distinguish from a real event. - responses: - "200": - description: Flag state restored to the baseline. - "404": - description: Not implemented; the TCK falls back to `/start`. - "501": - description: Not implemented; the TCK falls back to `/start`. - - /healthz: - get: - tags: [health] - operationId: health - summary: "[OPTIONAL] Readiness of the control API" - description: | - Reports whether the control API is ready to accept commands. - - **Fallback when not implemented.** Readiness defaults to "the control - port accepts a TCP connection", which the TCK establishes with a - Testcontainers listening-port wait strategy before the first scenario. A - `404` here is therefore not a failure, and the reference implementation - does not serve this path. - - Note this reports the health of the **control API**, not of the backend. - The backend is deliberately unhealthy during outage scenarios while the - control API must stay reachable — otherwise the TCK could not end the - outage. - responses: - "200": - description: Control API ready. - content: - application/json: - schema: - $ref: "#/components/schemas/Health" - "404": - description: Not implemented; readiness falls back to a TCP port check. - "503": - description: Control API not ready yet. - -components: - schemas: - - Health: - type: object - properties: - status: - type: string - enum: [ok] - description: Present and equal to `ok` when the control API is ready. - required: [status] - - Error: - type: object - properties: - message: - type: string - description: Human-readable explanation. Never interpreted by the TCK. - required: [message] diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/errors.feature b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/errors.feature deleted file mode 100644 index 0346df3da..000000000 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/errors.feature +++ /dev/null @@ -1,80 +0,0 @@ -Feature: Provider error handling - - # Every scenario here asserts the same three-part contract, because all three parts matter and - # providers routinely get one of them wrong: - # - # 1. the code default is returned — an application must keep working, - # 2. the correct error code is reported — an application must be able to tell what went wrong, - # 3. nothing is thrown — an unhandled exception from a flag evaluation is never acceptable. - # - # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. - - Background: - Given a stable provider - - Scenario Outline: Requesting the wrong type returns the code default - # The full non-numeric mismatch matrix. Numeric coercion is a separate question and is covered - # by the @strict-numeric-typing scenarios below, because "is 0.5 an integer?" has a defensible - # wrong answer whereas "is a string a boolean?" does not. - Given a -flag with key "" and a default value "" - When the flag was evaluated with details - Then the resolved details value should be "" - And the reason should be "ERROR" - And the error-code should be "TYPE_MISMATCH" - And no exception should have been thrown - - Examples: a string flag requested as something else - | key | requested | default | - | string-flag | Boolean | false | - | string-flag | Integer | 1 | - | string-flag | Float | 0.1 | - | wrong-flag | Boolean | false | - - Examples: a boolean flag requested as something else - | key | requested | default | - | boolean-flag | String | fallback | - | boolean-flag | Integer | 1 | - | boolean-flag | Float | 0.1 | - - Examples: a numeric flag requested as a non-numeric type - | key | requested | default | - | integer-flag | Boolean | false | - | integer-flag | String | fallback | - | float-flag | Boolean | false | - | float-flag | String | fallback | - - @object - Scenario Outline: Requesting a structured flag as a scalar returns the code default - Given a -flag with key "object-flag" and a default value "" - When the flag was evaluated with details - Then the resolved details value should be "" - And the reason should be "ERROR" - And the error-code should be "TYPE_MISMATCH" - And no exception should have been thrown - - Examples: - | requested | default | - | Boolean | false | - | String | fallback | - | Integer | 1 | - | Float | 0.1 | - - @strict-numeric-typing - Scenario: A float flag is not silently narrowed to an integer - # 'float-flag' resolves to 0.5. Narrowing that to an integer would lose information - # silently, so it must be reported as a type mismatch rather than rounded. - Given a Integer-flag with key "float-flag" and a default value "1" - When the flag was evaluated with details - Then the resolved details value should be "1" - And the reason should be "ERROR" - And the error-code should be "TYPE_MISMATCH" - And no exception should have been thrown - - Scenario: An unknown flag key returns the code default - # 'missing-flag' is deliberately absent from the canonical flag set. - Given a String-flag with key "missing-flag" and a default value "fallback" - When the flag was evaluated with details - Then the resolved details value should be "fallback" - And the reason should be "ERROR" - And the error-code should be "FLAG_NOT_FOUND" - And no exception should have been thrown diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/evaluation.feature b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/evaluation.feature deleted file mode 100644 index e89f174a5..000000000 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/evaluation.feature +++ /dev/null @@ -1,59 +0,0 @@ -Feature: Provider flag evaluation - - # Verifies that a provider maps backend responses onto typed resolution details correctly. - # - # This does NOT test the backend's evaluation logic. Every flag in the canonical set resolves - # to its default variant with no targeting involved, so what is under test is purely the - # provider's mapping of a backend response to a value, a variant and a reason. - # - # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. - - Background: - Given a stable provider - - Scenario Outline: Resolve values with variant and reason - Given a -flag with key "" and a default value "" - When the flag was evaluated with details - Then the resolved details value should be "" - And the variant should be "" - And the reason should be "" - And the error-code should be "" - And no exception should have been thrown - - Examples: - | key | type | default | value | variant | reason | - | boolean-flag | Boolean | false | true | on | STATIC | - | string-flag | String | bye | hi | greeting | STATIC | - | integer-flag | Integer | 1 | 10 | ten | STATIC | - | float-flag | Float | 0.1 | 0.5 | half | STATIC | - - Scenario: An integer flag resolves as an integer - # Paired with the float scenario below and with the narrowing scenario in errors.feature. - # Together they pin down that the two numeric types stay distinct rather than both being - # funnelled through one numeric representation. - Given a Integer-flag with key "integer-flag" and a default value "1" - When the flag was evaluated with details - Then the resolved details value should be "10" - And the error-code should be "" - And no exception should have been thrown - - Scenario: A float flag resolves as a float - Given a Float-flag with key "float-flag" and a default value "0.1" - When the flag was evaluated with details - Then the resolved details value should be "0.5" - And the error-code should be "" - And no exception should have been thrown - - @object - Scenario: Resolve a structured value - Given a Object-flag with key "object-flag" and a default value "{}" - When the flag was evaluated with details - Then the variant should be "template" - And the reason should be "STATIC" - And the error-code should be "" - And no exception should have been thrown - And the resolved object value should contain - | key | type | value | - | showImages | Boolean | true | - | title | String | Check out these pics! | - | imagesPerPage | Integer | 100 | diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/events.feature b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/events.feature deleted file mode 100644 index 00e7e5ef6..000000000 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/events.feature +++ /dev/null @@ -1,42 +0,0 @@ -@events -Feature: Provider events - - # Verifies that a provider notices changes in its backend and both signals them and acts on - # them. Signalling alone is not enough: a configuration-change event that is not followed by - # a changed evaluation result is a lie, so each scenario asserts the event AND the behaviour. - # - # Outages here are simulated inside the running stack via the control API. No container is - # ever stopped or restarted — see the invariant in openapi/control-api.yaml. - - Background: - Given a stable provider - - @configuration-change - Scenario: A configuration change is signalled and applied - Given a String-flag with key "changing-flag" and a default value "unset" - And a change event handler - When the flag was evaluated with details - And the resolved value is remembered - And the flag was modified - Then the change event handler should have been executed - And the flag should be part of the event payload - When the flag was evaluated with details - Then the resolved details value should have changed - And no exception should have been thrown - - @stale - Scenario: Losing the backend makes the provider stale, regaining it makes it ready again - Given a ready event handler - And a stale event handler - When a ready event was fired - And the connection is lost - Then the stale event handler should have been executed - And the client should be in stale state - When the connection is restored - Then the ready event handler should have been executed - And the client should be in ready state - - # Deliberately NOT covered here: whether a stale provider keeps serving last-known values - # during the outage. That is caching behaviour, which depends on whether the provider holds a - # local copy of the ruleset, and it belongs behind the @caching capability once those - # scenarios are written. See the "Known gaps" section of the README. diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/lifecycle.feature b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/lifecycle.feature deleted file mode 100644 index 256164106..000000000 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/features/lifecycle.feature +++ /dev/null @@ -1,33 +0,0 @@ -@events -Feature: Provider lifecycle - - # Verifies the two terminal outcomes of provider initialisation: reaching READY against a - # healthy backend, and settling into ERROR against one that cannot be reached. - # - # The failure case matters more than it looks. A provider that blocks forever, or throws out - # of provider registration, takes the host application down with it — so the requirement is - # not merely that initialisation fails, but that it fails observably and promptly. - - Scenario: A provider reaching its backend becomes ready - Given a stable provider - And a ready event handler - Then the ready event handler should have been executed - And the client should be in ready state - - @unavailable - Scenario: A provider that cannot reach its backend reports an error - Given a unavailable provider - And a error event handler - Then the error event handler should have been executed within 10000ms - And the client should be in error state - - @unavailable - Scenario: A provider that cannot reach its backend still returns code defaults - Given a unavailable provider - And a error event handler - And a Boolean-flag with key "boolean-flag" and a default value "false" - Then the error event handler should have been executed within 10000ms - When the flag was evaluated with details - Then the resolved details value should be "false" - And the reason should be "ERROR" - And no exception should have been thrown diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/flag_data/canonical-flags.json b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/flag_data/canonical-flags.json deleted file mode 100644 index 343b3ae52..000000000 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/flag_data/canonical-flags.json +++ /dev/null @@ -1,82 +0,0 @@ -{ - "$comment": [ - "The canonical flag set the TCK's feature files assume. A backend under test MUST serve an", - "equivalent set under the configuration named 'default'.", - "", - "Expressed in the flagd flag-definition format because that is the only widely implemented", - "vendor-neutral format today. The format is not what matters — the keys, types, variant", - "names and resolved values are. Seed them however your backend seeds flags.", - "", - "Two things are load-bearing and easy to get wrong:", - " * 'missing-flag' MUST NOT exist. Its absence is what the FLAG_NOT_FOUND scenario tests.", - " * No flag here has targeting rules. Every scenario expects reason STATIC, because the TCK", - " tests the provider's mapping of a response, not the backend's evaluation logic." - ], - "flags": { - "boolean-flag": { - "state": "ENABLED", - "variants": { - "on": true, - "off": false - }, - "defaultVariant": "on" - }, - "string-flag": { - "state": "ENABLED", - "variants": { - "greeting": "hi", - "parting": "bye" - }, - "defaultVariant": "greeting" - }, - "integer-flag": { - "state": "ENABLED", - "variants": { - "one": 1, - "ten": 10 - }, - "defaultVariant": "ten" - }, - "float-flag": { - "state": "ENABLED", - "variants": { - "tenth": 0.1, - "half": 0.5 - }, - "defaultVariant": "half" - }, - "object-flag": { - "state": "ENABLED", - "variants": { - "empty": {}, - "template": { - "showImages": true, - "title": "Check out these pics!", - "imagesPerPage": 100 - } - }, - "defaultVariant": "template" - }, - "wrong-flag": { - "$comment": "A string flag, evaluated as a boolean by the TYPE_MISMATCH scenario.", - "state": "ENABLED", - "variants": { - "one": "uno", - "two": "dos" - }, - "defaultVariant": "one" - }, - "changing-flag": { - "$comment": [ - "The flag POST /change mutates. The TCK asserts only that its resolved value differs", - "after the change, so which of the two variants you start from does not matter." - ], - "state": "ENABLED", - "variants": { - "foo": "foo", - "bar": "bar" - }, - "defaultVariant": "foo" - } - } -} From d6de5dc90478ea87950a37a6be89bc0d1f150d11 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 14:40:15 +0200 Subject: [PATCH 07/46] feat(provider-tck): add the @lifecycle capability lifecycle.feature was gated by @events, which was wrong in both directions. An SDK dispatches PROVIDER_READY around initialize for any provider (openfeature/provider/_registry.py), so a provider that declares @events passes the readiness scenario without demonstrating anything -- a NoOpProvider passes it identically. The gate made the scenario vacuous for exactly the providers it admitted. Conversely a stateless provider such as OFREP has a real initialisation to verify but no event stream of its own to declare @events for, so the gate shut it out of a scenario it should be held to. The spec revision pinned by the submodule retags the feature to @lifecycle and adds the capability to Appendix F. Add the matching enum member; plugin.py registers the marker by iterating the enum, so nothing else changes. Neither in-memory self-test declares it. They have no backend to reach, so their readiness scenario was passing vacuously too, and a skip with a reason is the honest outcome. 54 passed, 9 skipped, 2 xfailed. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 14 ++++++++++++- .../contrib/tools/provider_tck/capability.py | 21 +++++++++++++++++++ .../tests/test_controllable_conformance.py | 5 +++++ .../tests/test_in_memory_conformance.py | 8 +++++++ 4 files changed, 47 insertions(+), 1 deletion(-) diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index af37eda19..527360405 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -87,6 +87,7 @@ SKIPPED provider does not declare capability @stale. | Capability | Tag | Meaning | | --- | --- | --- | +| `Capability.LIFECYCLE` | `@lifecycle` | reaches its backend during initialisation, observably and promptly | | `Capability.EVENTS` | `@events` | emits lifecycle events at all | | `Capability.STALE` | `@stale` | enters `STALE` and emits `PROVIDER_STALE` on backend loss | | `Capability.CONFIGURATION_CHANGE` | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | @@ -96,6 +97,13 @@ SKIPPED provider does not declare capability @stale. | `Capability.TARGETING` | `@targeting` | reserved; no scenarios yet | | `Capability.CACHING` | `@caching` | reserved; no scenarios yet | +`@lifecycle` and `@events` are deliberately separate, and the split matters in both directions. An +SDK dispatches `PROVIDER_READY` around `initialize` for *any* provider, so a provider declaring only +`@events` passes the readiness scenario without demonstrating anything — a `NoOpProvider` passes it +identically. Meanwhile a stateless provider has a real initialisation to verify but no event stream +of its own to declare `@events` for, and gating on `@events` shut it out of a scenario it should be +held to. + Untagged scenarios are mandatory and always run. `capabilities` defaults to everything — narrow it rather than widening it: start from the default, run the suite, and remove only what your provider genuinely cannot do. @@ -205,11 +213,15 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_in_process_control` | `InProcessControl` | pins what the Gherkin cannot assert about itself | ``` -56 passed, 7 skipped, 2 xfailed +54 passed, 9 skipped, 2 xfailed ``` No Docker, no network, under a second. +Neither in-memory suite declares `@lifecycle`, so the three lifecycle scenarios are skipped in both. +That is the point: with no backend to reach, they would pass without testing anything — which is +what they did while the feature was gated on `@events`. + ## Known gaps - **Evaluation context passthrough is unverifiable.** The scenarios build evaluation contexts but diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py index 044908648..0444352ba 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py @@ -25,6 +25,27 @@ class Capability(str, Enum): Scenarios with no capability tag are mandatory and always run. """ + LIFECYCLE = "lifecycle" + """Provider reaches its backend during initialisation, observably and promptly. + + Deliberately separate from :attr:`EVENTS`, because the two are independent in + both directions. + + An SDK dispatches ``PROVIDER_READY`` around ``initialize`` for *any* + provider, so a provider that declares ``EVENTS`` passes the readiness + scenario without demonstrating anything -- a ``NoOpProvider`` passes it + identically. Gating on ``EVENTS`` therefore made the scenario vacuous for + exactly the providers that declared it. + + Conversely a stateless provider -- one that resolves every flag with a fresh + request and holds nothing between them -- has a real initialisation to + verify while having no event stream of its own to declare ``EVENTS`` for. + Gating on ``EVENTS`` shut it out of a scenario it should be held to. + + Declare it if initialisation actually contacts the backend and its outcome, + success or failure, is observable to the application. + """ + EVENTS = "events" """Provider emits lifecycle events at all, at minimum ``PROVIDER_READY``.""" diff --git a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py index 3ebd240ec..f127243c6 100644 --- a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py @@ -33,6 +33,11 @@ def tck_config() -> TckConfig: connection to lose, and ``InProcessControl`` does not implement ``ConnectionControl``. ``CONFIGURATION_CHANGE`` is what this suite adds over the plain in-memory one, and it is the whole point of it. + + ``LIFECYCLE`` stays undeclared for the same reason as in + ``test_in_memory_conformance``: there is no backend to reach during + initialisation, so the readiness scenario would pass here without testing + anything. It did exactly that while the feature was gated on ``@events``. """ control = InProcessControl() return TckConfig( diff --git a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py index de0250223..648ba05a8 100644 --- a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py @@ -85,6 +85,14 @@ def tck_config() -> TckConfig: operation the control cannot perform. * ``TARGETING`` and ``CACHING`` -- omitted because no scenario carries their tags yet, so leaving them out skips nothing. + * ``LIFECYCLE`` -- omitted because there is no backend to reach. The + capability asserts that initialisation actually contacts a backend and + that the outcome is observable; this provider's ``initialize`` is a no-op + and the SDK dispatches ``PROVIDER_READY`` around it regardless, so the + readiness scenario would pass here without testing anything. It passed + vacuously while the feature was gated on ``EVENTS``, which is precisely + the failure mode the split of ``@lifecycle`` from ``@events`` exists to + end. A skip with a reason is the honest outcome. """ return TckConfig( name="in-memory", From 51b6d3db45485e8d2f257041b1139e535c36bf70 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 09:47:24 +0200 Subject: [PATCH 08/46] refactor(provider-tck): rename @strict-numeric-typing to @numeric-coercion The capability vocabulary and the spec submodule pin belong to this PR, so the rename does too. It was written on the report branch, which is a sibling of the flagd and OFREP adoptions rather than an ancestor -- so the adoptions could not see it, and renaming their references there would have broken them against this base. Moving it down is what lets every branch above share one vocabulary. `Capability.STRICT_NUMERIC_TYPING` becomes `Capability.NUMERIC_COERCION`, marker `numeric-coercion`, and the submodule moves to dc4d7ae8 so the executed feature files carry the renamed tag. That bump also brings two unrelated spec changes: the lifecycle readiness scenario is renamed, and control-api.yaml gains the requirement that POST /start not return until the seeded state is served. The framing is corrected at the same time, because it was wrong rather than merely stale. Both the README and the capability's own docstring asserted that "the specification requires TYPE_MISMATCH when the requested type cannot be satisfied" and concluded that not declaring the capability was "an admission of a known bug". OpenFeature defines one numeric type deliberately -- `number` is "of unspecified type or size", and differentiating integers from floats is an optional language idiom -- so no requirement governs this, and the second claim followed from the first. The rule tested here is borrowed from flagd's numeric coercion ADR, which is scoped to flagd's own implementations; a provider behaving differently is not violating the specification. The gap in the provider contract is open-feature/spec#430, and flagd's own instance is open-feature/flagd#1996. That also makes the capability genuinely optional rather than a concession to a defect, which is the opposite of what the old text said. The report branch's own files stay with it: test_report.py does not exist here, and neither do the `not_applicable` and `known_deviations` configuration fields the rename also touched. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 28 ++++++++--- tools/openfeature-provider-tck/spec | 2 +- .../contrib/tools/provider_tck/capability.py | 47 +++++++++++++------ .../tests/test_controllable_conformance.py | 2 +- .../tests/test_in_memory_conformance.py | 2 +- 5 files changed, 57 insertions(+), 24 deletions(-) diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index 527360405..040141f7a 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -82,7 +82,7 @@ suite at all, so `pytest.skip` carries the reason into the report: ``` SKIPPED provider does not declare capability @stale. - Declared: @events @object @strict-numeric-typing + Declared: @events @numeric-coercion @object ``` | Capability | Tag | Meaning | @@ -93,7 +93,7 @@ SKIPPED provider does not declare capability @stale. | `Capability.CONFIGURATION_CHANGE` | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | | `Capability.OBJECT` | `@object` | supports structured flag values | | `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | -| `Capability.STRICT_NUMERIC_TYPING` | `@strict-numeric-typing` | does not coerce between integer and float | +| `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | | `Capability.TARGETING` | `@targeting` | reserved; no scenarios yet | | `Capability.CACHING` | `@caching` | reserved; no scenarios yet | @@ -108,11 +108,25 @@ Untagged scenarios are mandatory and always run. `capabilities` defaults to ever rather than widening it: start from the default, run the suite, and remove only what your provider genuinely cannot do. -`@strict-numeric-typing` deserves a note, because unlike the others it is **not** an optional -feature. The specification requires `TYPE_MISMATCH` when the requested type cannot be satisfied, and -narrowing `0.5` to `0` loses information silently. It is a capability only so a provider with the -defect can adopt today and see the gap reported explicitly rather than being unable to adopt at all. -Not declaring it is an admission of a known bug. +`@numeric-coercion` deserves a note, because it is the one capability here that **the specification +does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of +unspecified type or size", and languages **may** differentiate between integers and floats "as idioms +dictate" — so no requirement says what a provider must do when a value does not fit the accessor it +was asked through. That gap is [open-feature/spec#430](https://github.com/open-feature/spec/issues/430). + +The rule this tag is tested against is therefore **borrowed, not normative**: lossless coercion is +permitted, lossy coercion must fail — `10.0` requested as an integer must succeed, `0.5` must not. +It comes from flagd's +[numeric coercion ADR](https://github.com/open-feature/flagd/blob/main/docs/architecture-decisions/numeric-coercion.md), +which is scoped to flagd's own implementations, and the tag carries that name — it was +`@strict-numeric-typing` — because two vocabularies for one observable property is worse than one +borrowed name. **A provider that behaves differently is not violating the specification**, so +withholding this capability may be a deliberate choice as readily as a defect. + +Only the lossy half is tested. The canonical flag set has no integral float to ask the lossless half +of, so a provider that wrongly rejects `10.0` as an integer still passes; adding one changes the flag +set for every language at once. Appendix F records that as an open gap, together with a second one: +the width of a language's integer accessor — 64-bit against 32-bit — is not modelled at all. ## Controlling the backend diff --git a/tools/openfeature-provider-tck/spec b/tools/openfeature-provider-tck/spec index dfa16586d..dc4d7ae8d 160000 --- a/tools/openfeature-provider-tck/spec +++ b/tools/openfeature-provider-tck/spec @@ -1 +1 @@ -Subproject commit dfa16586d91ca020ef1b3b82a7c972d833ff8f29 +Subproject commit dc4d7ae8df1c664f82a4adf46cd43812980c0da3 diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py index 0444352ba..6021b461c 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py @@ -61,20 +61,39 @@ class Capability(str, Enum): UNAVAILABLE_INIT = "unavailable" """Provider reports an error state promptly against a backend it cannot reach.""" - STRICT_NUMERIC_TYPING = "strict-numeric-typing" - """Provider keeps the integer and float types distinct instead of coercing between them. - - Unlike every other entry here this is not an optional feature. The - specification requires a provider to report ``TYPE_MISMATCH`` when the - requested type cannot be satisfied, and narrowing ``0.5`` to ``0`` to satisfy - an integer request loses information silently -- the worst failure mode a - feature flag has, because the application sees a plausible value and no - error at all. - - It is a capability only so that a provider with this defect can adopt the - suite today and see the gap reported as an explicit skip, rather than being - unable to adopt at all. Not declaring it is an admission of a known bug, not - a design choice. Declare it as soon as the provider is fixed. + NUMERIC_COERCION = "numeric-coercion" + """Provider coerces between integer and float only when lossless, else ``TYPE_MISMATCH``. + + This is the one entry here that **the specification does not define**. + OpenFeature has a single numeric type on purpose -- ``number`` is "a numeric + value of unspecified type or size", and languages *may* differentiate between + integers and floats "as idioms dictate" -- so no requirement says what a + provider must do when a value does not fit the accessor it was asked through. + That gap is `open-feature/spec#430 + `_. + + The rule this capability is tested against is therefore **borrowed, not + normative**: lossless coercion is permitted, lossy coercion must fail. An + integral float such as ``10.0`` requested as an integer must succeed; ``0.5`` + must not. It comes from flagd's `numeric coercion ADR + `_, + which is scoped to flagd's own implementations, and the tag carries that name + -- it was ``@strict-numeric-typing`` -- because two vocabularies for one + observable property is worse than one borrowed name. + + **A provider that behaves differently is not violating the specification.** + So this is genuinely optional, rather than optional as a concession to a + defect: withholding it may be a deliberate choice as readily as a known bug. + Where it is a bug, say so -- a report's ``knownDeviations`` is for exactly + that, and flagd's instance is tracked as `open-feature/flagd#1996 + `_. + + Only the lossy half has a scenario. The canonical flag set has no integral + float to ask the lossless half of, and adding one changes the flag set for + every language at once, so a provider that wrongly rejects ``10.0`` as an + integer still passes. Appendix F records that as an open gap, along with a + second one: the width of a language's integer accessor is not modelled here + at all. """ TARGETING = "targeting" diff --git a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py index f127243c6..77b3c3d0b 100644 --- a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py @@ -48,7 +48,7 @@ def tck_config() -> TckConfig: Capability.EVENTS, Capability.CONFIGURATION_CHANGE, Capability.OBJECT, - Capability.STRICT_NUMERIC_TYPING, + Capability.NUMERIC_COERCION, }, ) diff --git a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py index 648ba05a8..9b9e4a191 100644 --- a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py @@ -101,7 +101,7 @@ def tck_config() -> TckConfig: capabilities={ Capability.EVENTS, Capability.OBJECT, - Capability.STRICT_NUMERIC_TYPING, + Capability.NUMERIC_COERCION, }, ) From 40b803ae3c9e421101ab326d3b1e840eebb4e87a Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 11:46:41 +0200 Subject: [PATCH 09/46] feat(provider-tck): complete the declaration an adoption makes A TckConfig is two things. It is the configuration a run needs, and it is the set of claims an adopter makes about their provider -- which is what turns a skipped scenario from a hole in the run into a recorded answer. The second role was incomplete, and the gaps were all the same shape: something an adopter has to be able to say that the vocabulary gave them no way to say. `not_applicable={Capability.X: "why"}` is a capability that *cannot* hold rather than one the adopter chose not to declare. The suite treats the two identically, because the scenarios are skipped either way, but collapsing them misrepresents whole languages: @numeric-coercion is unsatisfiable in JavaScript, which has no integer type, and recording that as a choice shows every JavaScript provider as declining something none of them can have. `known_deviations` acknowledges a gap against something the specification does not treat as optional, with somewhere it is tracked. An acknowledgement and not an excuse: the scenario still fails and the suite still fails with it. What it adds is that the gap was known rather than a surprise. And a capability is now either declarable or reserved. @targeting and @caching gate no scenario, so declaring one cannot be verified, cannot produce a skip, and says only that something was claimed and nothing examined -- so declaring one is refused at construction, where the adopter's own code is still on the stack to say which line to fix. The default is DECLARABLE_CAPABILITIES rather than the whole enum, because "declare everything, then narrow it" is the advice and therefore the one place a reserved tag gets declared by accident: that is how one implementation's published report came to assert both of them. `capability_for_tag` and `control_api` are here for a consumer that is not. A reporter outside this package has to tell a capability-gating tag from a merely organisational one, and has to be able to ask a control how it drove the backend. Nothing in this commit calls either; that is the point. Two branches sit on this one, and neither should be able to change what the other compiles against. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 39 ++- .../contrib/tools/provider_tck/__init__.py | 8 +- .../contrib/tools/provider_tck/capability.py | 56 +++- .../contrib/tools/provider_tck/config.py | 160 ++++++++- .../contrib/tools/provider_tck/control.py | 14 + .../contrib/tools/provider_tck/inprocess.py | 10 + .../tests/test_declaration.py | 312 ++++++++++++++++++ 7 files changed, 578 insertions(+), 21 deletions(-) create mode 100644 tools/openfeature-provider-tck/tests/test_declaration.py diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index 040141f7a..5779ab06b 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -94,8 +94,8 @@ SKIPPED provider does not declare capability @stale. | `Capability.OBJECT` | `@object` | supports structured flag values | | `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | | `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | -| `Capability.TARGETING` | `@targeting` | reserved; no scenarios yet | -| `Capability.CACHING` | `@caching` | reserved; no scenarios yet | +| `Capability.TARGETING` | `@targeting` | reserved; **not declarable** — no scenarios yet | +| `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | `@lifecycle` and `@events` are deliberately separate, and the split matters in both directions. An SDK dispatches `PROVIDER_READY` around `initialize` for *any* provider, so a provider declaring only @@ -104,9 +104,18 @@ identically. Meanwhile a stateless provider has a real initialisation to verify of its own to declare `@events` for, and gating on `@events` shut it out of a scenario it should be held to. -Untagged scenarios are mandatory and always run. `capabilities` defaults to everything — narrow it -rather than widening it: start from the default, run the suite, and remove only what your provider -genuinely cannot do. +Untagged scenarios are mandatory and always run. `capabilities` defaults to every *declarable* +capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from +the default, run the suite, and remove only what your provider genuinely cannot do. + +A reserved capability is documented so the vocabulary has a place for it once scenarios exist, and +until then it **must not be declared**. Nothing carries the tag, so declaring it cannot be verified, +cannot produce a skip, and tells anyone reading the declaration only that something was claimed and +nothing examined. `TckConfig` raises if you name one in `capabilities` or in `not_applicable`, and +`DECLARABLE_CAPABILITIES` excludes them — which is the case that matters, because "every capability +except X" is how a reserved tag gets declared by accident rather than by decision. One +implementation's published conformance report asserts `@targeting` and `@caching` for exactly that +reason. `@numeric-coercion` deserves a note, because it is the one capability here that **the specification does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of @@ -128,6 +137,23 @@ of, so a provider that wrongly rejects `10.0` as an integer still passes; adding set for every language at once. Appendix F records that as an open gap, together with a second one: the width of a language's integer accessor — 64-bit against 32-bit — is not modelled at all. +### Declaring more than a capability set + +Two further fields on `TckConfig` say things a capability set cannot, and both are declarations +rather than switches: neither changes which scenarios run or what they assert. + +`not_applicable={Capability.X: "why"}` is for a capability that *cannot* hold rather than one you +chose not to declare. The suite treats the two identically — the scenarios are skipped either way, +with the reason — but collapsing them misrepresents a provider, and whole languages with it: +`@numeric-coercion` is unsatisfiable in JavaScript because the language has no integer type, and +recording that as a choice would show every JavaScript provider as declining something none of them +can have. Declining an optional feature is a choice; an impossibility is not. + +`known_deviations=(KnownDeviation(issue=..., summary=...),)` acknowledges a gap against something the +specification does *not* treat as optional, with somewhere it is tracked. It is an acknowledgement +and not an excuse: the scenario still fails and the suite still fails with it. What the declaration +adds is that the gap was known rather than a surprise. + ## Controlling the backend `BackendControl` is the single seam between the scenarios and whatever manipulates the backend. Step @@ -225,9 +251,10 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_in_memory_conformance` | the SDK's `InMemoryProvider` | reference adoption for a backend-less provider | | `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | | `test_in_process_control` | `InProcessControl` | pins what the Gherkin cannot assert about itself | +| `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | ``` -54 passed, 9 skipped, 2 xfailed +70 passed, 9 skipped, 2 xfailed ``` No Docker, no network, under a second. diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py index 8b615296d..b3dcb2e82 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py @@ -49,8 +49,8 @@ def tck_config(): import importlib.resources -from .capability import ALL_CAPABILITIES, Capability -from .config import TckConfig +from .capability import DECLARABLE_CAPABILITIES, RESERVED_CAPABILITIES, Capability +from .config import KnownDeviation, TckConfig from .control import ( BackendControl, ConnectionControl, @@ -64,13 +64,15 @@ def tck_config(): ) __all__ = [ - "ALL_CAPABILITIES", "CHANGING_FLAG_KEY", + "DECLARABLE_CAPABILITIES", + "RESERVED_CAPABILITIES", "BackendControl", "Capability", "ConnectionControl", "ControllableInMemoryProvider", "InProcessControl", + "KnownDeviation", "TckConfig", "UnsupportedControlError", "canonical_flag_set", diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py index 6021b461c..8be2b2403 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py @@ -97,29 +97,62 @@ class Capability(str, Enum): """ TARGETING = "targeting" - """Reserved. No scenario carries this tag: targeting is backend evaluation logic.""" + """Reserved, and **not declarable**. No scenario carries this tag: targeting + is backend evaluation logic.""" CACHING = "caching" - """Reserved; no scenario carries this tag yet.""" + """Reserved, and **not declarable**. No scenario carries this tag yet.""" @property def tag(self) -> str: """Return the Gherkin tag, with its leading at-sign, that gates this capability.""" return f"@{self.value}" + @property + def reserved(self) -> bool: + """Whether this capability exists in the vocabulary but gates no scenario.""" + return self in RESERVED_CAPABILITIES + def __str__(self) -> str: return self.tag -ALL_CAPABILITIES: frozenset[Capability] = frozenset(Capability) -"""Every capability the TCK recognises. +RESERVED_CAPABILITIES: frozenset[Capability] = frozenset( + {Capability.TARGETING, Capability.CACHING} +) +"""Capabilities that exist in the vocabulary and gate no scenario. + +They are documented so the vocabulary has a place for them when scenarios exist, +and until then they **must not be declared** and must not appear in a conformance +report's declaration. Nothing carries the tag, so declaring it cannot be +verified, cannot produce a skip, and tells a reader of the report only that +something was claimed and nothing examined. + +Listed once, here, and read everywhere else -- by +:data:`DECLARABLE_CAPABILITIES`, by :attr:`Capability.reserved` and by the +validation in :class:`~.config.TckConfig` -- so that the set and the rule cannot +drift apart. +""" + +DECLARABLE_CAPABILITIES: frozenset[Capability] = ( + frozenset(Capability) - RESERVED_CAPABILITIES +) +"""Every capability an adoption may declare: the vocabulary minus the reserved tags. A reasonable starting point for a new adoption: declare everything, run the -suite, and remove only what the provider genuinely cannot do. Narrowing from the -full set surfaces gaps; widening towards it hides them. +suite, and remove only what the provider genuinely cannot do. Narrowing from this +set surfaces gaps; widening towards it hides them. + +It excludes the reserved capabilities rather than spanning the whole enum, and it +is named for what it is rather than for "all", because the declare-everything +convenience is exactly how a reserved tag reaches a report by accident: an +adopter writing "every capability except X" picks up every reserved tag on the +way past, which is how one implementation came to report ``@targeting`` and +``@caching`` as declared without anyone deciding to claim them. """ _BY_MARKER: dict[str, Capability] = {c.value: c for c in Capability} +_BY_TAG: dict[str, Capability] = {c.tag: c for c in Capability} def capability_for_marker(name: str) -> Capability | None: @@ -129,3 +162,14 @@ def capability_for_marker(name: str) -> Capability | None: the canonical feature files carry organisational tags freely. """ return _BY_MARKER.get(name) + + +def capability_for_tag(tag: str) -> Capability | None: + """Map a Gherkin tag, leading at-sign included, onto the capability it gates. + + The tag form rather than the marker form because that is what the + conformance report carries: the report records a scenario's tags as the + feature files spell them, and deciding whether a failure counts against a + capability means reading them back. + """ + return _BY_TAG.get(tag) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py index 77b783cdb..384702669 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py @@ -2,15 +2,16 @@ from __future__ import annotations -from collections.abc import Callable, Collection, Iterable +import typing +from collections.abc import Callable, Collection, Iterable, Mapping, Sequence from dataclasses import dataclass, field from openfeature.provider import FeatureProvider -from .capability import ALL_CAPABILITIES, Capability +from .capability import DECLARABLE_CAPABILITIES, Capability from .control import BackendControl -__all__ = ["ProviderFactory", "TckConfig"] +__all__ = ["KnownDeviation", "ProviderFactory", "TckConfig"] ProviderFactory = Callable[[], FeatureProvider] """Creates the provider under test. @@ -24,6 +25,46 @@ DEFAULT_READY_TIMEOUT = 30.0 +@dataclass(frozen=True) +class KnownDeviation: + """A gap the provider is known to have, acknowledged rather than hidden. + + Distinct from an undeclared capability, which is a choice, and from a + not-applicable one, which is impossible: this is a defect against something + the specification does not treat as optional, with the gap tracked + somewhere. + + It changes nothing about how the suite runs. The scenario still fails, and + the results payload still reports it as failed -- a report that softened a + failure into a footnote would hide exactly what the acknowledgement exists to + keep visible. What this adds is the acknowledgement itself, in the envelope, + so that a consumer can tell a known and tracked gap from a surprise. + """ + + issue: str + """Where the gap is tracked. A URI, because the schema requires one.""" + + summary: str + """What is wrong, for a person reading a comparison page.""" + + capability: Capability | None = None + """The capability the deviation concerns, when it maps to one. + + Left out for a deviation against a mandatory scenario, which belongs to no + capability -- which is the common case, since a capability a provider fails + is usually one it should not have declared. + """ + + def as_json(self) -> dict[str, typing.Any]: + document: dict[str, typing.Any] = { + "issue": self.issue, + "summary": self.summary, + } + if self.capability is not None: + document["capability"] = self.capability.tag + return document + + @dataclass(frozen=True) class TckConfig: """Everything the TCK needs to test one provider. @@ -79,7 +120,7 @@ class TckConfig: skipped with the reason reported. """ - capabilities: Collection[Capability] = field(default=ALL_CAPABILITIES) + capabilities: Collection[Capability] = field(default=DECLARABLE_CAPABILITIES) """Which optional parts of the provider contract this provider supports. Typed as a ``Collection`` rather than a ``frozenset`` so that the obvious @@ -88,8 +129,39 @@ class TckConfig: construction, so a list, a set or a generator all behave identically. Scenarios tagged with an undeclared capability are reported as skipped with - the reason, never as passed. Defaults to everything; narrow it rather than - widening it. + the reason, never as passed. Defaults to every *declarable* capability -- + :data:`~.capability.DECLARABLE_CAPABILITIES`, which excludes the reserved + tags no scenario carries -- and narrowing it surfaces gaps where widening + towards it hides them. + + Naming a reserved capability here is rejected at construction rather than + passed into a report. See :data:`~.capability.RESERVED_CAPABILITIES`. + """ + + not_applicable: Mapping[Capability, str] = field(default_factory=dict) + """Capabilities that cannot hold for this provider, each with a reason. + + Kept apart from simply leaving a capability out of :attr:`capabilities`, + because the two are different claims and collapsing them misrepresents whole + languages: ``@numeric-coercion`` is unsatisfiable in JavaScript because + the language has no integer type, and reporting that as a choice would show + every JavaScript provider as missing something none of them can have. + + Scenarios behind a not-applicable capability are skipped exactly as an + undeclared one's are -- the gate makes no distinction, and neither does the + results payload. The difference is recorded once, here, and reaches the + report's declaration. + + Where the impossibility is a property of the language rather than of the + provider it belongs in the capability documentation rather than in every + report, so this is for provider-specific cases. + """ + + known_deviations: Sequence[KnownDeviation] = () + """Gaps this provider is known to have, with each one tracked somewhere. + + An acknowledgement, not an excuse: the scenarios still fail and the results + payload still says so. See :class:`KnownDeviation`. """ event_timeout: float = DEFAULT_EVENT_TIMEOUT @@ -138,6 +210,47 @@ def __post_init__(self) -> None: f"the Capability enum" ) + # Normalised the same way, so a dict literal keyed by Capability is what + # an adopter writes and a plain mapping is what everything else reads. + object.__setattr__(self, "not_applicable", dict(self.not_applicable)) + object.__setattr__(self, "known_deviations", tuple(self.known_deviations)) + + stray = [c for c in self.not_applicable if not isinstance(c, Capability)] + if stray: + problems.append( + f"unknown capabilities {stray!r} in not_applicable: capabilities are " + f"the members of the Capability enum" + ) + + both = sorted( + capability.tag + for capability in self.not_applicable + if isinstance(capability, Capability) and capability in self.capabilities + ) + if both: + problems.append( + f"capabilities and not_applicable both claim {' '.join(both)}: a " + f"capability is either declared or impossible, and a report saying " + f"both leaves a consumer to guess which" + ) + + problems.extend( + reserved_problems(self.capabilities, self.not_applicable.keys()) + ) + + unreasoned = sorted( + capability.tag + for capability, reason in self.not_applicable.items() + if isinstance(capability, Capability) + and (not isinstance(reason, str) or not reason.strip()) + ) + if unreasoned: + problems.append( + f"not_applicable gives no reason for {' '.join(unreasoned)}: " + f"'impossible for this provider' is only useful to a reader who is " + f"told why, and the report schema requires the reason" + ) + if ( Capability.UNAVAILABLE_INIT in self.capabilities and self.new_unavailable_provider is None @@ -174,6 +287,41 @@ def sorted_capabilities(self) -> list[str]: return sorted(c.tag for c in self.capabilities) +def reserved_problems(*named: Iterable[Capability]) -> list[str]: + """Refuse a reserved capability named anywhere in a configuration. + + A reserved capability gates no scenario, so naming it cannot be verified + either way: declaring it claims something nothing examined, and calling it + not-applicable records an impossibility about a question that was never + asked. Either would reach the report's declaration, which the schema + forbids. + + Refused rather than dropped quietly. The adopter wrote it down and meant + something by it, so a configuration silently different from the one they + wrote is worse than one that will not build -- and construction is where + their own code is still on the stack to say which line to fix. The + alternative, a warning, is a line of CI output nobody reads while an + untested capability goes on being asserted in a published report, which is + how this got into one in the first place. + """ + reserved = sorted( + capability.tag + for group in named + for capability in group + if isinstance(capability, Capability) and capability.reserved + ) + if not reserved: + return [] + declarable = " ".join(sorted(c.tag for c in DECLARABLE_CAPABILITIES)) + return [ + f"reserved capabilities {' '.join(sorted(set(reserved)))} cannot be declared " + f"or called not-applicable: no scenario carries them, so the claim cannot be " + f"verified, cannot produce a skip, and would tell a reader of the report only " + f"that something was claimed and nothing examined. The declarable " + f"capabilities, which is what DECLARABLE_CAPABILITIES holds, are {declarable}" + ] + + def capabilities_of(values: Iterable[Capability]) -> frozenset[Capability]: """Convenience for building a capability set from any iterable.""" return frozenset(values) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py index 0e83e5bd2..e92dd18d4 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py @@ -73,6 +73,20 @@ def change_flag(self) -> None: def description(self) -> str: """A short description of what is being controlled, for messages a human reads.""" + # OPTIONAL: ``control_api`` + # + # A control may also offer a ``control_api`` property returning ``"http"`` + # for the normative HTTP control API, or ``"in-process"`` for the narrow + # allowance made for providers with no backend. The conformance report + # records it, so that a claim of in-process control by a provider that does + # have a backend can be treated with the suspicion it deserves. + # + # It is deliberately not a member of this protocol. Adding one would make + # every existing control incomplete for the sake of one string, and there is + # nothing useful the TCK can do with a control that has not said: it cannot + # tell from the outside whether a control spoke HTTP or reached into the + # process, so the field is simply omitted. See ``report.control_api_of``. + @typing.runtime_checkable class ConnectionControl(typing.Protocol): diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py index 1d69254c6..11586e0ff 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py @@ -63,6 +63,16 @@ def __init__(self) -> None: def description(self) -> str: return "in-process control of an in-memory provider" + @property + def control_api(self) -> str: + """Report how this backend was driven, for the conformance report. + + ``in-process`` is the narrow allowance for providers with no backend, + which is exactly what this control exists for. A provider that does have + a backend and reports this is claiming something it should not. + """ + return "in-process" + def new_provider(self) -> FeatureProvider: """Create the provider for the scenario about to run, at the baseline. diff --git a/tools/openfeature-provider-tck/tests/test_declaration.py b/tools/openfeature-provider-tck/tests/test_declaration.py new file mode 100644 index 000000000..d6cc3ddac --- /dev/null +++ b/tools/openfeature-provider-tck/tests/test_declaration.py @@ -0,0 +1,312 @@ +"""What an adoption may declare about its provider, and what it may not. + +A ``TckConfig`` is two things at once. It is the configuration a run needs, and +it is a *declaration*: the set of claims an adopter makes about the provider, +which is what turns a skipped scenario from a hole in the run into a recorded +answer. Everything checked here belongs to the second role, so none of it is +observable in the pass/fail of a suite -- which is exactly why it is pinned by +tests of its own rather than by the conformance suites. + +The declaration vocabulary is public because something outside this package has +to read it back. A reporter deciding whether a skip was legitimate needs the tag +lookup and the reserved set; a comparison page needs to tell a declined +capability from an impossible one. None of those consumers is here, and the API +is complete for them anyway -- a follow-up that adds one should widen nothing. + +The one property that makes "reserved" mean anything is checked against the +packaged assets rather than asserted: a reserved tag is reserved because no +canonical scenario carries it, and that stops being true the moment the spec +adds one. +""" + +from __future__ import annotations + +import re +import typing +from pathlib import Path + +import pytest + +from openfeature.contrib.tools.provider_tck import ( + DECLARABLE_CAPABILITIES, + RESERVED_CAPABILITIES, + BackendControl, + Capability, + InProcessControl, + KnownDeviation, + TckConfig, + features_path, +) +from openfeature.contrib.tools.provider_tck.capability import ( + capability_for_marker, + capability_for_tag, +) + + +class _StubControl: + """A control that says nothing it is not obliged to say. + + Which includes ``control_api``: the property is documented as optional, and + a control leaving it out has to remain a ``BackendControl``. + """ + + def prepare_scenario(self) -> None: ... + + def change_flag(self) -> None: ... + + @property + def description(self) -> str: + return "a stub" + + +def _config(**overrides: typing.Any) -> TckConfig: + """A configuration that is valid but declares nothing in particular.""" + settings: dict[str, typing.Any] = { + "name": "stub", + "control": _StubControl(), + "new_provider": lambda: None, + "capabilities": frozenset(), + } + settings.update(overrides) + return TckConfig(**settings) + + +def _canonical_tags() -> set[str]: + """Every tag the packaged feature files carry, at any level. + + Read off tag lines only. A tag is the whole of the line it appears on in + Gherkin, which is what tells one apart from the same word written in a + comment -- ``events.feature`` mentions ``@caching`` in prose, saying where + those scenarios will go once they exist. + """ + tags: set[str] = set() + for feature in sorted(Path(features_path()).glob("*.feature")): + for line in feature.read_text(encoding="utf-8").splitlines(): + stripped = line.strip() + if stripped.startswith("@"): + tags.update(re.findall(r"@[\w-]+", stripped)) + return tags + + +# -- the vocabulary ---------------------------------------------------------- + + +def test_every_capability_is_either_declarable_or_reserved() -> None: + """Two sets, one enum, and no member in both or neither. + + ``DECLARABLE_CAPABILITIES`` is derived from ``RESERVED_CAPABILITIES`` rather + than listed beside it, so this is really a check that the derivation is the + one the documentation promises. + """ + assert frozenset(Capability) == DECLARABLE_CAPABILITIES | RESERVED_CAPABILITIES + assert not DECLARABLE_CAPABILITIES & RESERVED_CAPABILITIES + assert RESERVED_CAPABILITIES, "the whole rule is vacuous if nothing is reserved" + + for capability in Capability: + assert capability.reserved is (capability in RESERVED_CAPABILITIES) + + +def test_a_reserved_capability_is_one_no_canonical_scenario_carries() -> None: + """The fact the rule rests on, read off the assets rather than asserted. + + "Reserved" claims that nothing carries the tag, so declaring it cannot be + verified. If the specification adds a scenario for ``@targeting``, that + stops being true and this fails -- which is the moment the capability should + become declarable, and the moment somebody has to notice. + """ + carried = _canonical_tags() + for capability in RESERVED_CAPABILITIES: + assert capability.tag not in carried, ( + f"{capability.tag} is no longer reserved: the canonical assets now " + f"carry it, so it can be verified and should be declarable" + ) + for capability in DECLARABLE_CAPABILITIES: + assert capability.tag in carried, ( + f"{capability.tag} is declarable but no canonical scenario carries " + f"it, so declaring it would be a claim nothing examines" + ) + + +def test_a_tag_maps_onto_the_capability_it_gates() -> None: + """The lookup a reporter outside this package needs, in the tag form. + + The tag form rather than the marker form, because that is the form a + scenario's tags are recorded in: deciding whether a skip was legitimate + means reading them back as the feature files spell them. Nothing in this + package calls it -- it is exported for the consumer that does. + """ + for capability in Capability: + assert capability_for_tag(capability.tag) is capability + assert capability_for_marker(capability.value) is capability + + # An organisational tag gates nothing, and must not be mistaken for a + # capability: the feature files carry them freely. + assert capability_for_tag("@smoke") is None + assert capability_for_tag("events") is None, "the at-sign is part of the tag" + + +# -- declaring a capability set ---------------------------------------------- + + +def test_the_default_is_every_declarable_capability() -> None: + """And so cannot pick up a reserved tag on the way past. + + "Declare everything, then narrow it" is the advice, which makes the default + the one place a reserved tag would otherwise get declared by accident. One + implementation's published report asserts ``@targeting`` and ``@caching`` + for precisely that reason. + """ + # Not routed through ``_config``, which narrows the set: the field default is + # the whole point of this one. It needs an unavailable-provider factory + # because ``@unavailable`` is declarable, so the default declares it. + settings: dict[str, typing.Any] = { + "name": "stub", + "control": _StubControl(), + "new_provider": lambda: None, + "new_unavailable_provider": lambda: None, + } + declared = TckConfig(**settings).capabilities + assert declared == DECLARABLE_CAPABILITIES + for capability in RESERVED_CAPABILITIES: + assert capability not in declared + + +def test_a_capability_set_is_normalised_however_it_was_written() -> None: + """A list, a set or a generator all arrive as the same frozenset.""" + expected = frozenset({Capability.EVENTS, Capability.OBJECT}) + written = [ + [Capability.EVENTS, Capability.OBJECT, Capability.EVENTS], + {Capability.EVENTS, Capability.OBJECT}, + (c for c in (Capability.EVENTS, Capability.OBJECT)), + ] + for capabilities in written: + assert _config(capabilities=capabilities).capabilities == expected + + +def test_something_that_is_not_a_capability_is_refused() -> None: + with pytest.raises(ValueError, match="unknown capabilities"): + _config(capabilities={"events"}) + + +def test_a_reserved_capability_cannot_be_declared() -> None: + """A tag no scenario carries is a claim nothing can check, so it is refused. + + At construction rather than at the point something reads the declaration: + the adopter wrote it down and meant something by it, so a configuration + silently different from the one they wrote is worse than one that will not + build -- and construction is where their own code is still on the stack to + say which line to fix. + """ + for reserved in RESERVED_CAPABILITIES: + with pytest.raises(ValueError, match=f"reserved capabilities {reserved.tag}"): + _config(capabilities={Capability.EVENTS, reserved}) + with pytest.raises(ValueError, match=f"reserved capabilities {reserved.tag}"): + _config(not_applicable={reserved: "no scenario asks"}) + + +def test_the_refusal_says_what_may_be_declared_instead() -> None: + """A message that names the rule and not just the violation.""" + with pytest.raises(ValueError) as raised: + _config(capabilities={Capability.TARGETING}) + message = str(raised.value) + assert "DECLARABLE_CAPABILITIES" in message + for capability in DECLARABLE_CAPABILITIES: + assert capability.tag in message + + +# -- declaring an impossibility ---------------------------------------------- + + +def test_not_applicable_is_normalised_and_keeps_its_reasons() -> None: + """Written as a dict literal keyed by ``Capability``; read as a mapping.""" + config = _config(not_applicable={Capability.NUMERIC_COERCION: "no integer type"}) + assert dict(config.not_applicable) == { + Capability.NUMERIC_COERCION: "no integer type" + } + + +def test_a_capability_cannot_be_both_declared_and_impossible() -> None: + """The two are different claims, and a declaration asserting both says neither.""" + with pytest.raises(ValueError, match="both claim @events"): + _config( + capabilities={Capability.EVENTS}, + not_applicable={Capability.EVENTS: "a reason"}, + ) + + +def test_a_not_applicable_capability_must_say_why() -> None: + """A reason is required: "impossible for this provider" is useless without one.""" + for empty in ("", " ", "\n"): + with pytest.raises(ValueError, match="no reason for @stale"): + _config(not_applicable={Capability.STALE: empty}) + + +def test_something_that_is_not_a_capability_cannot_be_not_applicable() -> None: + with pytest.raises(ValueError, match="in not_applicable"): + _config(not_applicable={"stale": "a reason"}) + + +# -- acknowledging a gap ----------------------------------------------------- + + +def test_a_known_deviation_carries_its_capability_only_when_it_has_one() -> None: + """The common case has none: a mandatory scenario belongs to no capability. + + Omitted rather than null, because the field is the answer to "which + capability does this concern" and there is not always one. + """ + issue = "https://github.com/open-feature/python-sdk/issues/619" + mandatory = KnownDeviation(issue=issue, summary="a boolean satisfies an Integer") + assert mandatory.as_json() == {"issue": issue, "summary": mandatory.summary} + + attributed = KnownDeviation( + issue=issue, + summary="a lossy float satisfies an Integer", + capability=Capability.NUMERIC_COERCION, + ) + assert attributed.as_json() == { + "issue": issue, + "summary": attributed.summary, + "capability": Capability.NUMERIC_COERCION.tag, + } + + +def test_known_deviations_are_normalised_and_change_nothing_about_the_run() -> None: + """Declared as any sequence; read as a tuple. + + And that is all they do. A deviation is an acknowledgement, not a licence: + nothing here makes a scenario pass, skip, or be collected differently, which + is why a suite declaring one still fails on it. + """ + deviation = KnownDeviation(issue="https://example.invalid/1", summary="a gap") + config = _config(known_deviations=[deviation]) + assert config.known_deviations == (deviation,) + assert _config().known_deviations == () + assert _config(known_deviations=[deviation]).capabilities == _config().capabilities + + +# -- saying how the backend is driven ---------------------------------------- + + +def test_a_control_need_not_say_how_it_drives_the_backend() -> None: + """``control_api`` is documented as optional, and means it. + + Making it a member of ``BackendControl`` would make every existing control + incomplete for the sake of one string, and there is nothing the suite can do + with the answer: it cannot tell from the outside whether a control spoke + HTTP or reached into the process. + """ + quiet = _StubControl() + assert isinstance(quiet, BackendControl) + assert not hasattr(quiet, "control_api") + + +def test_in_process_control_says_it_is_in_process() -> None: + """The narrow allowance, and the control that exists to take it. + + A provider that does have a backend and reports this is claiming something + it should not, which is only detectable if the honest case says so plainly. + """ + control = InProcessControl() + assert isinstance(control, BackendControl) + assert control.control_api == "in-process" From 230bd40b6049dd549f63810f7f98f18686058cb9 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 11:55:01 +0200 Subject: [PATCH 10/46] feat(provider-tck): let an adopter add their own scenarios to the suite A provider is rarely only a provider. flagd has `fractional` targeting, another vendor has a proprietary rollout rule, and pinning those used to mean a second harness beside the conformance suite: a second backend lifecycle, a second set of fixtures, a second thing to keep working. An adopter's scenarios now run inside the canonical suite instead -- same session, same provider registration, same backend control. Almost nothing was needed to make that happen, because pytest already scans: it collects `conftest.py` on its own and pytest-bdd resolves step definitions through the fixture system, so a step an adopter writes beside their test module is already in scope for the scenarios generated into it. The only thing pytest cannot find by itself is the feature files, because the canonical ones live inside the installed distribution. `feature_paths()` returns both -- the packaged assets, and a `tck-extensions` directory beside the calling module if there is one -- so an adoption gains one call and no configuration: scenarios(*feature_paths()) An extension must never be able to stand in for a canonical scenario. Java's suite found that a same-named feature file in a second classpath root replaced the canonical one outright and the run went green having asked the adopter's questions; Python has a narrower route to the same place, because pytest-bdd names a feature file by its parent directory joined to its own name and `tck-extensions/features/errors.feature` therefore arrives under the uri the canonical `errors.feature` already occupies. So the uri a feature file is identified by is derived from where the file is: `features/` for the packaged assets and nothing else, `extensions/` for anything below a `tck-extensions` directory -- the same prefix the Go and JavaScript suites mount extensions under, so a consumer holding reports from several languages applies one rule. The two cases the derivation cannot rule out are reported rather than raised, because the scenarios are the adopter's to run and it is publishing them as the specification's that has to be refused: a file of the adopter's own that would reach the reserved `features/` prefix, and two feature files that would share one uri, which nothing recording a run can hold because it keeps one copy of a feature file per uri. Whatever refuses to publish is not here. The derivation and both problems are public, and the self-test reads a generated adoption back through pytest's own JUnit XML rather than through a conformance report, which this package does not write. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 77 ++- .../contrib/tools/provider_tck/__init__.py | 31 +- .../contrib/tools/provider_tck/extensions.py | 296 +++++++++++ .../contrib/tools/provider_tck/plugin.py | 7 +- .../tests/test_extensions.py | 488 ++++++++++++++++++ 5 files changed, 877 insertions(+), 22 deletions(-) create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py create mode 100644 tools/openfeature-provider-tck/tests/test_extensions.py diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index 5779ab06b..3669b0e18 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -32,7 +32,7 @@ from pytest_bdd import scenarios from openfeature.contrib.tools.provider_tck import ( Capability, TckConfig, - features_path, + feature_paths, ) @@ -47,7 +47,7 @@ def tck_config(): ) -scenarios(features_path()) +scenarios(*feature_paths()) ``` There is **no `conftest.py` to write and nothing to import for the steps**. The step definitions @@ -70,6 +70,72 @@ different timescales — a streaming provider sees a configuration change in mil polls every 30 seconds may need most of a poll interval. Set it to comfortably exceed your worst-case detection latency, or the suite reports timeouts that are really just impatience. +## Adding your own scenarios + +A provider is rarely only a provider. flagd has `fractional` targeting, another vendor has a +proprietary rollout rule, and the behaviour of those is as worth pinning as the contract they sit on +top of. Verifying them used to mean a second harness: a second backend lifecycle, a second set of +fixtures, a second thing to keep working. + +Put them in the same run instead. Create a directory named `tck-extensions` beside the module that +calls `scenarios()`, and write step definitions for whatever is new in a `conftest.py` beside it: + +``` +tests/ +├── conftest.py # your step definitions +├── test_conformance.py # the fixture and the one call, unchanged +└── tck-extensions/ + └── fractional.feature +``` + +```python +# conftest.py +from pytest_bdd import then + +from openfeature.contrib.tools.provider_tck import TckState + + +@then("the fractional rule splits the population") +def fractional_splits(tck_state: TckState) -> None: ... +``` + +That is the whole of it — **no registration, no option and no new argument**. pytest collects +`conftest.py` on its own, pytest-bdd resolves steps through the fixture system, and the canonical +step vocabulary is in scope in your feature file beside your own steps. `tck_state` is the same +per-scenario state the canonical steps use, so your scenario runs against the provider the suite +registered, in the same backend lifecycle, with the same reset between scenarios. + +The one thing pytest cannot find by itself is the feature files, because the canonical ones are +inside the installed distribution rather than in your repository. `feature_paths()` returns both: + +```python +scenarios(*feature_paths()) +``` + +That line does not change when you add an extension, and it is the only difference from +`scenarios(features_path())` — which still works and still sees only the canonical set. An adopter +with no `tck-extensions` directory runs exactly what they ran before: same scenarios, same count. + +### Your scenarios cannot stand in for ours + +Every feature file carries a uri, and it is how a canonical scenario is told from an adopter's: +canonical files are the ones under the `features/` prefix and yours are under `extensions/` — the +prefix Go and JavaScript mount theirs under too, so a consumer holding conformance reports from +several languages applies one rule. The prefix is derived from where a file *is*, not from what the +runner called it, and `extensions.py` reports two cases that derivation cannot rule out: + +- **A feature file of yours under the reserved `features/` prefix.** Handing `scenarios()` a + directory of your own named `features` is the one route left to a canonical-looking uri. +- **Two feature files that would share one uri.** A record of what ran holds one copy of a feature + file per uri, so the second file's scenarios would be attributed to the first file's. + +This is not hypothetical. Java's suite found that a same-named feature file in a second classpath +root *replaced* the canonical one, and the run went green having asked the adopter's questions +instead of the specification's — the worst outcome available to a conformance suite. The Python +route to the same place is narrower and just as quiet: pytest-bdd names a feature file by its parent +directory joined to its own name, so `tck-extensions/features/errors.feature` arrives under the uri +the canonical `errors.feature` already occupies. + ## Capabilities Not every provider implements every optional part of the contract. Each scenario exercising an @@ -252,12 +318,15 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | | `test_in_process_control` | `InProcessControl` | pins what the Gherkin cannot assert about itself | | `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | +| `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity | ``` -70 passed, 9 skipped, 2 xfailed +84 passed, 9 skipped, 2 xfailed ``` -No Docker, no network, under a second. +No Docker and no network. The conformance suites take under a second; `test_extensions` takes most +of the rest, because the properties it checks are properties of a whole pytest session and it runs a +generated adoption in a subprocess to check them. Neither in-memory suite declares `@lifecycle`, so the three lifecycle scenarios are skipped in both. That is the point: with no backend to reach, they would pass without testing anything — which is diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py index b3dcb2e82..ed361a34b 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py @@ -16,7 +16,7 @@ Capability, InProcessControl, TckConfig, - features_path, + feature_paths, ) @pytest.fixture(scope="session") @@ -29,11 +29,14 @@ def tck_config(): capabilities={Capability.EVENTS, Capability.OBJECT}, ) - scenarios(features_path()) + scenarios(*feature_paths()) ``scenarios()`` is pytest-bdd's own, called directly rather than wrapped: it injects the generated tests into the *calling module* by walking the stack, so a convenience wrapper around it would deposit them inside this package instead. +:func:`~.extensions.feature_paths` is the canonical assets plus a +``tck-extensions`` directory beside the calling module, if there is one -- see +:mod:`~.extensions`. The step definitions arrive through this package's pytest plugin, so there is nothing to import for them and no ``conftest.py`` to write. Everything else -- @@ -56,16 +59,23 @@ def tck_config(): ConnectionControl, UnsupportedControlError, ) +from .extensions import ( + EXTENSIONS_DIRECTORY, + feature_paths, + features_path, +) from .inprocess import InProcessControl from .provider import ( CHANGING_FLAG_KEY, ControllableInMemoryProvider, canonical_flag_set, ) +from .state import TckState __all__ = [ "CHANGING_FLAG_KEY", "DECLARABLE_CAPABILITIES", + "EXTENSIONS_DIRECTORY", "RESERVED_CAPABILITIES", "BackendControl", "Capability", @@ -74,10 +84,12 @@ def tck_config(): "InProcessControl", "KnownDeviation", "TckConfig", + "TckState", "UnsupportedControlError", "canonical_flag_set", "canonical_flags_json", "control_api_spec", + "feature_paths", "features_path", ] @@ -103,21 +115,6 @@ def tck_config(): _PACKAGE = "openfeature.contrib.tools.provider_tck" -def features_path() -> str: - """Return the directory holding the canonical feature files. - - Packaged with this distribution, so a consumer needs no submodule and no - particular directory layout. Hand it to pytest-bdd's ``scenarios()``, which - accepts an absolute path:: - - scenarios(features_path()) - - pytest-bdd generates one test per scenario -- and one per row of a Scenario - Outline -- so failures name a scenario and ``-k`` selects one as usual. - """ - return str(importlib.resources.files(_PACKAGE) / "features") - - def canonical_flags_json() -> str: """Return the canonical flag set as raw JSON, in the flagd flag-definition format. diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py new file mode 100644 index 000000000..e5796d64e --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py @@ -0,0 +1,296 @@ +"""Where the scenarios come from: the canonical set, plus whatever an adopter adds. + +A provider is rarely only a provider. flagd has ``fractional`` targeting, another +vendor has a proprietary rollout rule, and the behaviour of those is as worth +pinning as the contract they sit on top of. Verifying them used to mean standing +up a second harness: a second backend lifecycle, a second set of fixtures, a +second thing to keep working. The canonical suite ran, then something else ran, +and nothing tied the two together. + +So an adopter's own scenarios run **inside** the canonical suite instead -- +against the same provider instance, in the same backend lifecycle, with the same +step vocabulary available. Almost nothing is needed to make that happen, because +pytest already scans. It collects ``conftest.py`` on its own and pytest-bdd +resolves step definitions through the fixture system, so a step defined in the +adopter's ``conftest.py`` -- or in the test module itself -- is in scope for the +scenarios ``scenarios()`` generates there. The only thing pytest cannot find by +itself is the feature files, which is what this module finds: a directory named +``tck-extensions`` beside the adopter's test module. + +That leaves one line, and it is the same line whether or not there are +extensions:: + + scenarios(*feature_paths()) + +**An extension can never stand in for a canonical scenario.** The two are told +apart by the uri each feature file is identified by, and this module derives that +uri from where the file *is* rather than taking what the runner offers: + +* ``features/…`` is the packaged canonical assets, and nothing else; +* ``extensions/…`` is a discovered extension, whatever the adopter's own + directory layout under ``tck-extensions`` looks like. + +The derivation is not decoration. pytest-bdd names a feature file by its parent +directory joined to its own name, so ``tck-extensions/features/errors.feature`` +arrives as ``features/errors.feature`` -- the same uri as a canonical file. A +record of what ran holds one copy of a feature file per uri, so the second file +is never read and its scenarios are attributed to the first one's or to nothing +at all. Java hit the same thing by a different route: a same-named feature file +in a second classpath root replaced the canonical one outright and the suite went +green having run the adopter's version. + +The derivation is public, and the two problems it cannot rule out are reported +rather than raised, because the consumer of all of this is a conformance report +and that is not written here. Appendix F requires a report to say which scenarios +ran; nothing else can tell an adopter's question from the specification's. +""" + +from __future__ import annotations + +import importlib.resources +import inspect +import typing +from pathlib import Path + +__all__ = [ + "CANONICAL_DIRECTORY", + "EXTENSIONS_DIRECTORY", + "EXTENSIONS_URI_PREFIX", + "canonical_root", + "collision_problem", + "extension_root", + "feature_paths", + "features_path", + "is_canonical", + "is_canonical_uri", + "reserved_prefix_problem", + "uri_collisions", + "uri_for", +] + +_PACKAGE = "openfeature.contrib.tools.provider_tck" + +CANONICAL_DIRECTORY = "features" +"""The packaged directory the canonical feature files live in. + +Also the uri prefix they are identified by, which is why it is reserved: anyone +reading ``features/errors.feature`` is entitled to assume it is the +specification's file rather than a local one that happened to land in a directory +of that name. +""" + +EXTENSIONS_DIRECTORY = "tck-extensions" +"""Where an adopter puts feature files of their own, beside their test module. + +Deliberately not ``features``: a directory sharing the canonical name is how an +extension comes to occupy a canonical file's identity, and a convention that +cannot collide is worth more than one that reads slightly better. The name is +the one Java's TCK scans for on the classpath, so an adopter who ships a provider +in both languages puts the same directory in both repositories. +""" + +EXTENSIONS_URI_PREFIX = "extensions" +"""The uri prefix an extension's scenarios are identified by. + +The Go and JavaScript suites mount extensions under the same prefix, so a +consumer holding reports from several languages applies one rule to tell an +adopter's scenario from the specification's. +""" + + +def features_path() -> str: + """Return the directory holding the canonical feature files. + + Packaged with this distribution, so a consumer needs no submodule and no + particular directory layout. This is the canonical set on its own; prefer + :func:`feature_paths`, which also picks up an adopter's own scenarios. + """ + return str(importlib.resources.files(_PACKAGE) / CANONICAL_DIRECTORY) + + +def feature_paths() -> tuple[str, ...]: + """Return every feature directory this adoption should run. + + The canonical set, always, and a ``tck-extensions`` directory beside the + calling module if there is one. Hand the result to pytest-bdd's + ``scenarios()``:: + + scenarios(*feature_paths()) + + That line does not change when an adopter adds an extension, which is what + makes adding one a matter of creating a directory rather than of configuring + anything. + + The calling module is located from the caller's frame, which is how + pytest-bdd locates it for ``scenarios()`` itself, so the two agree about + which module is adopting the suite. Call it from the test module rather than + from a helper: a helper's directory is what a helper would find. A caller + with no ``__file__`` -- an interactive session, an exec'd string -- gets the + canonical set alone. + """ + paths = [features_path()] + directory = _caller_directory() + if directory is not None: + extensions = extension_root(directory) + if extensions is not None: + paths.append(str(extensions)) + return tuple(paths) + + +def extension_root(module_directory: Path) -> Path | None: + """The extension directory beside a test module, or ``None`` if there is none. + + ``None`` rather than a path that contributes nothing, so that an adopter + without extensions hands ``scenarios()`` exactly what they handed it before: + same scenarios, same count, same report. + """ + candidate = module_directory / EXTENSIONS_DIRECTORY + return candidate if candidate.is_dir() else None + + +def canonical_root() -> Path | None: + """The packaged canonical features directory, as a real path. + + ``None`` if the assets are not on the filesystem -- an installation from a + zipimport, say. Everything built on this degrades to "cannot tell", which is + the honest answer and never a false accusation. + """ + try: + return _resolve(Path(features_path())) + except (OSError, TypeError): # pragma: no cover - assets outside a filesystem + return None + + +def is_canonical(path: Path) -> bool: + """Whether a feature file is one of the packaged canonical ones.""" + canonical = canonical_root() + return canonical is not None and _resolve(path).is_relative_to(canonical) + + +def is_canonical_uri(uri: str) -> bool: + """Whether a uri names a canonical feature file. + + The discriminator between a canonical scenario and an adopter's own wherever + it matters. Derived from the uri rather than carried beside it, so there is + no second fact to disagree with the first. + """ + return uri.startswith(f"{CANONICAL_DIRECTORY}/") + + +def uri_for(path: Path) -> str | None: + """The uri a feature file should be identified by. + + ``None`` when the file is neither canonical nor under an extension + directory, in which case the caller falls back to what pytest-bdd named it. + + Derived from the file's location rather than from pytest-bdd's + ``rel_filename``, which is the parent directory's name joined to the file's + own. That is what let ``tck-extensions/features/errors.feature`` present + itself as ``features/errors.feature``: the same uri as a canonical file, and + a record of what ran holds one copy of a feature file per uri. + """ + resolved = _resolve(path) + + canonical = canonical_root() + if canonical is not None and resolved.is_relative_to(canonical): + return _uri(Path(CANONICAL_DIRECTORY) / resolved.relative_to(canonical)) + + for parent in resolved.parents: + if parent.name == EXTENSIONS_DIRECTORY: + return _uri(Path(EXTENSIONS_URI_PREFIX) / resolved.relative_to(parent)) + return None + + +def reserved_prefix_problem(uri: str, path: Path) -> str | None: + """Report a feature file claiming the canonical uri prefix without being canonical. + + The one thing the naming convention cannot rule out on its own: an adopter + who hands ``scenarios()`` a directory of their own named ``features``. The + file is then named exactly as a canonical one would be, and a reader has no + way to tell that the specification did not write it. + + Returned rather than raised. The suite itself has no use for the answer -- + the scenarios run either way, and they are the adopter's to run -- so this is + for whatever writes a record of the run to refuse to publish one. + """ + if not is_canonical_uri(uri) or is_canonical(path): + return None + return ( + f"{uri} is not a canonical feature file -- it is {path} -- but it would be " + f"reported under the {CANONICAL_DIRECTORY}/ prefix, which is reserved for " + f"the packaged conformance assets. Move it into a directory named " + f"{EXTENSIONS_DIRECTORY} beside the test module, which feature_paths() " + f"finds on its own" + ) + + +def uri_collisions( + identified: typing.Iterable[tuple[str, Path]], +) -> dict[str, tuple[Path, ...]]: + """Feature files that would share one uri, keyed by that uri. + + Deriving the uri from the file's location removes the collision an adopter + is actually likely to hit, but it does not make one impossible. Two + extension roots contributing the same relative path to a single suite -- two + test modules sharing one ``tck_config`` from a conftest, each with a + ``tck-extensions/vendor.feature`` -- still land on ``extensions/vendor.feature`` + twice, and so does a ``tck-extensions`` directory nested inside another one. + + That has to be refused rather than resolved. A record of what ran holds one + copy of a feature file per uri, so the second file is never read: its + scenarios are attributed to the first file's where the names happen to + match, and go missing where they do not. The first is the silent form of + exactly the failure Java measured, and it is the one a consumer cannot + detect from the outside. + + Compared by resolved path, so the same file reached by two routes is one + file rather than a collision. + """ + files: dict[str, dict[Path, None]] = {} + for uri, path in identified: + files.setdefault(uri, {})[_resolve(path)] = None + return {uri: tuple(paths) for uri, paths in files.items() if len(paths) > 1} + + +def collision_problem(uri: str, paths: typing.Sequence[Path]) -> str: + """Say which files collided and what to do about it.""" + listed = ", ".join(str(path) for path in sorted(paths)) + return ( + f"{uri} is the uri of {len(paths)} different feature files -- {listed} -- " + f"and a record of what ran holds one copy of a feature file per uri, so " + f"one of them would be reported against the other's. Give them paths that " + f"differ below their {EXTENSIONS_DIRECTORY} directory" + ) + + +def _caller_directory() -> Path | None: + """The directory of the module two frames up, if it has a file.""" + frame = inspect.currentframe() + for _ in range(2): + if frame is None: # pragma: no cover - no Python frames to walk + return None + frame = frame.f_back + if frame is None: # pragma: no cover - called with no caller above + return None + file_name: typing.Any = frame.f_globals.get("__file__") + if not isinstance(file_name, str) or not file_name: + return None + return _resolve(Path(file_name)).parent + + +def _resolve(path: Path) -> Path: + try: + return path.resolve() + except OSError: # pragma: no cover - a path that cannot be resolved at all + return path + + +def _uri(path: Path) -> str: + """A relative path as a uri: slash-separated on every platform. + + These paths are assembled with ``pathlib``, so on Windows they arrive + backslash-separated. A uri is not, and the same string has to identify a + feature file wherever the suite ran or a run on Windows is not comparable + with one on Linux. + """ + return path.as_posix() diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py index b8b1a73b1..d4b66bb4b 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py @@ -4,7 +4,12 @@ all it takes for the step definitions to be available. pytest-bdd resolves steps through the fixture system and fixtures from an installed plugin are visible to every test, which is what keeps an adoption down to one fixture and one call to -:func:`tck_scenarios`. +``scenarios(*feature_paths())``. + +The same mechanism is what makes the suite extensible: a step an adopter defines +in their own ``conftest.py`` is resolved by the same fixture lookup as one this +plugin ships, so their scenarios need no glue and no second harness. See +:mod:`~.extensions`. """ from __future__ import annotations diff --git a/tools/openfeature-provider-tck/tests/test_extensions.py b/tools/openfeature-provider-tck/tests/test_extensions.py new file mode 100644 index 000000000..8d0b407e6 --- /dev/null +++ b/tools/openfeature-provider-tck/tests/test_extensions.py @@ -0,0 +1,488 @@ +"""What an adopter's own scenarios may and may not do. + +An adopter with provider-specific behaviour -- flagd's ``fractional`` targeting, a +proprietary rollout rule -- has to be able to pin it in the same run as the +contract it sits on top of, or they end up maintaining a second harness beside +the one the TCK gives them. So the properties checked here are the ones that make +that safe rather than merely possible: + +* an extension scenario runs **inside** the canonical suite -- same session, same + provider registration, same backend control -- with a step definition the + adopter wrote in their own ``conftest.py`` and nothing else registered; +* an adoption without extensions runs exactly what it ran before, scenario for + scenario and outcome for outcome; +* a feature file's identity comes from where the file is, so an extension cannot + take a canonical scenario's. + +The last is not hypothetical. Java's suite discovered a same-named feature file +in a second classpath root silently *replacing* the canonical one, and the run +went green having asked the adopter's questions instead of the specification's. +The Python route to the same place is narrower and just as quiet: pytest-bdd +names a feature file by its parent directory joined to its own name, so a file at +``tck-extensions/features/errors.feature`` arrives under the uri the canonical +``errors.feature`` already occupies. + +The first three are properties of how a whole session runs rather than of what a +function returns, so they are checked against real pytest sessions in +subprocesses, read back through pytest's own JUnit XML. Reading them back from a +conformance report would be circular here and impossible anyway: this package +writes none. +""" + +from __future__ import annotations + +import dataclasses +import subprocess +import sys +import xml.etree.ElementTree as ElementTree +from pathlib import Path + +import pytest + +from openfeature.contrib.tools.provider_tck import ( + EXTENSIONS_DIRECTORY, + feature_paths, + features_path, +) +from openfeature.contrib.tools.provider_tck.extensions import ( + CANONICAL_DIRECTORY, + EXTENSIONS_URI_PREFIX, + collision_problem, + extension_root, + is_canonical, + is_canonical_uri, + reserved_prefix_problem, + uri_collisions, + uri_for, +) + +CANONICAL_FEATURE = "errors.feature" +"""The canonical file the collision cases are written against, chosen because it +is the one whose scenarios an extension could most plausibly want to restate.""" + +VENDOR_SCENARIO = "A vendor rule resolves through the suite's own provider" + + +# -- the generated adoption -------------------------------------------------- + +_SUITE_MODULE = '''\ +"""A one-fixture adoption, generated so extensions can be checked end to end.""" + +import pytest +from pytest_bdd import scenarios + +from openfeature.contrib.tools.provider_tck import ( + Capability, + InProcessControl, + TckConfig, + feature_paths, + features_path, +) + + +@pytest.fixture(scope="session") +def tck_config(): + control = InProcessControl() + return TckConfig( + name="{name}", + control=control, + new_provider=control.new_provider, + capabilities={{Capability.EVENTS, Capability.OBJECT}}, + ) + + +scenarios({call}) +''' + +_EXTENSION_CALL = "*feature_paths()" +_CANONICAL_CALL = "features_path()" + +# The step the adopter writes, in the adopter's own conftest.py and nowhere else. +# It asks the TCK's own per-scenario state what happened, which is what makes +# "the same session and the same provider" checkable rather than asserted: a +# second harness would have a second provider, or none, and none of the canonical +# steps would have run. +_CONFTEST_MODULE = """\ +import pytest +from pytest_bdd import then + +from openfeature.contrib.tools.provider_tck import TckState + +DEVIATION = "[boolean-flag-Integer-1]" + + +@then("the vendor rule ran against the provider the suite registered") +def vendor_rule_ran(tck_state: TckState) -> None: + assert tck_state.client is not None, "no provider was registered" + assert tck_state.last is not None, "the canonical steps did not run here" + assert tck_state.last.value == "hi", tck_state.last + + +def pytest_collection_modifyitems(items): + for item in items: + if item.name.endswith(DEVIATION): + item.add_marker(pytest.mark.xfail(reason="python-sdk#619")) +""" + +# Deliberately reuses the canonical step vocabulary and adds exactly one step of +# its own, which is the shape an adopter's feature file actually takes. +_VENDOR_FEATURE = """\ +Feature: Vendor rules + + Background: + Given a stable provider + + Scenario: A vendor rule resolves through the suite's own provider + Given a String-flag with key "string-flag" and a default value "bye" + When the flag was evaluated with details + Then the resolved details value should be "hi" + And the vendor rule ran against the provider the suite registered +""" + + +# -- reading a run back ------------------------------------------------------ + + +@dataclasses.dataclass(frozen=True) +class Run: + """One subprocess run of a generated adoption.""" + + directory: Path + result: subprocess.CompletedProcess[str] + outcomes: dict[str, str] + + +def _outcomes(report: Path) -> dict[str, str]: + """Read a JUnit XML report into ``node id -> passed | failed | skipped``. + + pytest's own results format, because the question is what the session did + and pytest is the thing that knows. It records one ``testcase`` per test with + the file it came from, which is what lets two suites in one directory be + told apart. + """ + outcomes: dict[str, str] = {} + # Not untrusted input: the file is one pytest wrote seconds ago, in a + # temporary directory this test made, from a subprocess this test started. + root = ElementTree.parse(report).getroot() # noqa: S314 + for case in root.iter("testcase"): + statuses = { + "failure": "failed", + "error": "failed", + "skipped": "skipped", + } + status = "passed" + for tag, named in statuses.items(): + if case.find(tag) is not None: + status = named + break + node = f"{case.get('classname', '')}::{case.get('name', '')}" + outcomes[node] = status + return outcomes + + +def _run( + tmp_path_factory: pytest.TempPathFactory, + modules: dict[str, str], + features: dict[str, str] | None = None, +) -> Run: + """Write an adoption, run it in a subprocess, and read the results back. + + Both mappings are keyed by a path relative to the adoption directory, so a + case can put a module or a feature file wherever the property under test + needs it. + """ + directory = tmp_path_factory.mktemp("adoption") + for relative, body in {**modules, **(features or {})}.items(): + path = directory / Path(relative) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(body, encoding="utf-8") + + report = directory / "results.xml" + result = subprocess.run( # noqa: S603 + [ + sys.executable, + "-m", + "pytest", + "-q", + "-p", + "no:cacheprovider", + f"--junitxml={report}", + str(directory), + ], + capture_output=True, + text=True, + check=False, + ) + outcomes = _outcomes(report) if report.exists() else {} + return Run(directory=directory, result=result, outcomes=outcomes) + + +def _suite(name: str, call: str = _EXTENSION_CALL) -> str: + return _SUITE_MODULE.format(name=name, call=call) + + +@pytest.fixture(scope="module") +def adoption(tmp_path_factory: pytest.TempPathFactory) -> Run: + """One session running two adoptions of the same provider. + + ``before`` is the call an adopter wrote before any of this, + ``scenarios(features_path())``, which sees no extension however many are + lying beside it. ``after`` is ``scenarios(*feature_paths())`` with an + ordinary extension beside it. Having both in one run is what lets "an + extension adds and does not alter" be a comparison rather than a number + written down here. + + One session rather than two, because a subprocess pytest run is by far the + most expensive thing in this file and the two suites are independent: each + resolves its own ``TckConfig`` under its own OpenFeature domain. + """ + return _run( + tmp_path_factory, + { + "test_before.py": _suite("before", _CANONICAL_CALL), + "test_after.py": _suite("after", _EXTENSION_CALL), + "conftest.py": _CONFTEST_MODULE, + }, + {f"{EXTENSIONS_DIRECTORY}/vendor.feature": _VENDOR_FEATURE}, + ) + + +def _of(run: Run, module: str) -> dict[str, str]: + """The outcomes belonging to one of the generated suites, keyed by test name. + + JUnit XML names the module in dotted form, so the two generated suites in one + directory are told apart by the last segment of it. + """ + return { + node.split("::", 1)[1]: status + for node, status in run.outcomes.items() + if node.split("::", 1)[0].rsplit(".", 1)[-1] == module + } + + +def _contributed(run: Run) -> dict[str, str]: + """What the extension added: the tests ``after`` ran and ``before`` did not. + + Identified by difference rather than by name. pytest-bdd derives a test + function's name from the scenario name by a munging of its own -- an + apostrophe disappears where a space becomes an underscore -- and reproducing + that here would pin pytest-bdd's spelling rather than this package's + behaviour. + """ + before = _of(run, "test_before") + return { + name: status + for name, status in _of(run, "test_after").items() + if name not in before + } + + +# -- an extension runs inside the canonical suite ---------------------------- + + +def test_an_extension_scenario_runs_in_the_canonical_suite(adoption: Run) -> None: + """One suite, one session, both sets of scenarios in it. + + The extension scenario passes only if it reached the provider the suite + registered and the canonical steps ran in it, because that is what its own + step asserts -- so this is not a presentational fact about collection. It is + the same suite, which is the same provider registration and the same backend + control. + """ + assert adoption.result.returncode == 0, adoption.result.stdout + + contributed = _contributed(adoption) + assert len(contributed) == 1, adoption.outcomes + [(name, status)] = contributed.items() + assert "vendor_rule" in name, name + assert status == "passed" + + canonical = _of(adoption, "test_after") + assert len(canonical) > 1, "the canonical scenarios must have run too" + assert "passed" in canonical.values() + + +def test_the_extension_step_came_from_the_adopters_conftest(adoption: Run) -> None: + """Nothing was registered, imported or configured to make that step resolve. + + pytest collects ``conftest.py`` on its own and pytest-bdd resolves steps + through the fixture system, so a step defined beside the test module is in + scope for the scenarios generated into it. An unresolved step is a *failure* + rather than an omission, which is why asserting that the scenario passed is + enough to pin this. + """ + conftest = (adoption.directory / "conftest.py").read_text(encoding="utf-8") + assert "the vendor rule ran against the provider the suite registered" in conftest + assert set(_contributed(adoption).values()) == {"passed"} + + +# -- and changes nothing for an adopter who has none ------------------------- + + +def test_an_extension_adds_scenarios_and_alters_none(adoption: Run) -> None: + """``before`` is what an adopter ran before extensions existed. + + Every test it generated, ``after`` generated too, with the same outcome, and + the only difference between the two is the one scenario the extension added. + An adopter who has no extensions is the same comparison with the right-hand + side empty, which is what ``feature_paths()`` returning the canonical path + alone makes true by construction rather than by luck. + """ + before = _of(adoption, "test_before") + after = _of(adoption, "test_after") + assert before, "the baseline suite generated nothing" + + assert len(_contributed(adoption)) == 1 + assert {name: after[name] for name in before} == before + + +def test_features_path_sees_no_extension_however_many_are_beside_it( + adoption: Run, +) -> None: + """The older call still means exactly what it meant: the canonical set. + + Both generated suites sit in the same directory as the ``tck-extensions`` + directory, so the one that asks for ``features_path()`` is asking with an + extension in arm's reach and must still not see it. + """ + assert (adoption.directory / EXTENSIONS_DIRECTORY).is_dir() + assert set(_of(adoption, "test_before")) < set(_of(adoption, "test_after")) + + +# -- finding an adopter's feature files -------------------------------------- + + +def test_feature_paths_is_the_canonical_set_when_there_is_no_extension_directory() -> ( + None +): + """This test module has no ``tck-extensions`` beside it, and gets one path.""" + assert not (Path(__file__).parent / EXTENSIONS_DIRECTORY).exists() + assert feature_paths() == (features_path(),) + + +def test_an_extension_directory_counts_only_when_it_is_a_directory( + tmp_path: Path, +) -> None: + """``None`` rather than a path that contributes nothing. + + So that an adopter without extensions hands ``scenarios()`` exactly what + they handed it before -- and so that a *file* of that name, which pytest-bdd + would choke on, is not offered as a feature directory. + """ + assert extension_root(tmp_path) is None + + (tmp_path / EXTENSIONS_DIRECTORY).write_text("not a directory", encoding="utf-8") + assert extension_root(tmp_path) is None + + (tmp_path / EXTENSIONS_DIRECTORY).unlink() + (tmp_path / EXTENSIONS_DIRECTORY).mkdir() + assert extension_root(tmp_path) == tmp_path / EXTENSIONS_DIRECTORY + + +def test_the_canonical_features_are_found_inside_the_distribution() -> None: + """No submodule and no directory layout of the adopter's own.""" + packaged = Path(features_path()) + assert (packaged / CANONICAL_FEATURE).is_file() + assert is_canonical(packaged / CANONICAL_FEATURE) + assert not is_canonical(Path(__file__)) + + +# -- deriving the uri -------------------------------------------------------- + + +def test_the_canonical_assets_keep_the_reserved_prefix() -> None: + canonical = Path(features_path()) / CANONICAL_FEATURE + assert uri_for(canonical) == f"{CANONICAL_DIRECTORY}/{CANONICAL_FEATURE}" + assert is_canonical_uri(f"{CANONICAL_DIRECTORY}/{CANONICAL_FEATURE}") + assert reserved_prefix_problem(f"features/{CANONICAL_FEATURE}", canonical) is None + + +def test_an_extension_keeps_its_layout_below_the_extensions_prefix( + tmp_path: Path, +) -> None: + """Whatever the adopter's own directory layout under the root looks like. + + Including one that reproduces the canonical name, which is the collision the + derivation exists for: pytest-bdd would have called the second of these + ``features/errors.feature``. + """ + root = tmp_path / EXTENSIONS_DIRECTORY + assert uri_for(root / "vendor.feature") == "extensions/vendor.feature" + assert ( + uri_for(root / CANONICAL_DIRECTORY / CANONICAL_FEATURE) + == f"{EXTENSIONS_URI_PREFIX}/{CANONICAL_DIRECTORY}/{CANONICAL_FEATURE}" + ) + assert ( + uri_for(root / "a" / "b" / "vendor.feature") == "extensions/a/b/vendor.feature" + ) + assert not is_canonical_uri(f"extensions/features/{CANONICAL_FEATURE}") + + +def test_a_derived_uri_is_slash_separated_on_every_platform(tmp_path: Path) -> None: + """A uri identifies a feature file, so it cannot depend on where it ran. + + The paths these are built from are ``pathlib`` paths, which are + backslash-separated on Windows. A run there has to be comparable with one on + Linux, and it is not if the same file is identified two ways. + """ + nested = tmp_path / EXTENSIONS_DIRECTORY / "a" / "b" / "vendor.feature" + derived = uri_for(nested) + assert derived == "extensions/a/b/vendor.feature" + assert derived is not None and "\\" not in derived + + +def test_a_file_that_is_neither_is_left_to_pytest_bdd(tmp_path: Path) -> None: + """``None`` rather than a guess: the caller falls back to what the runner said.""" + assert uri_for(tmp_path / "loose.feature") is None + + +def test_a_local_file_under_the_reserved_prefix_is_a_problem(tmp_path: Path) -> None: + """The one route to a canonical-looking uri the convention cannot close. + + An adopter may still hand ``scenarios()`` a directory of their own named + ``features``, and its files are then named exactly as canonical ones would + be. Reported rather than raised: the scenarios are the adopter's to run, and + it is publishing them as the specification's that has to be refused. + """ + local = tmp_path / CANONICAL_DIRECTORY / "local.feature" + problem = reserved_prefix_problem(f"{CANONICAL_DIRECTORY}/local.feature", local) + assert problem is not None + assert EXTENSIONS_DIRECTORY in problem, "the message must say the fix" + assert str(local) in problem + + +def test_two_files_that_would_share_one_uri_are_reported(tmp_path: Path) -> None: + """Deriving the uri from the location narrows the collision; it does not end it. + + A ``tck-extensions`` directory nested inside another one reaches the same uri + as its namesake at the root, and so would two test modules sharing one + ``tck_config``. + """ + root = tmp_path / EXTENSIONS_DIRECTORY + nested = root / "nested" / EXTENSIONS_DIRECTORY / "vendor.feature" + collisions = uri_collisions( + [ + ("extensions/vendor.feature", root / "vendor.feature"), + ("extensions/vendor.feature", nested), + ] + ) + assert set(collisions) == {"extensions/vendor.feature"} + + problem = collision_problem( + "extensions/vendor.feature", collisions["extensions/vendor.feature"] + ) + assert "2 different feature files" in problem + assert EXTENSIONS_DIRECTORY in problem + + +def test_distinct_extension_paths_do_not_collide(tmp_path: Path) -> None: + """The layouts that are fine, including the one that only looks like a clash.""" + root = tmp_path / EXTENSIONS_DIRECTORY + assert not uri_collisions( + [ + ("extensions/vendor.feature", root / "vendor.feature"), + ("extensions/a/vendor.feature", root / "a" / "vendor.feature"), + # One file reached by two routes is one file, not a collision. + ("extensions/vendor.feature", root / "a" / ".." / "vendor.feature"), + ] + ) From e37196755fd3498355c6253329aa49ff7212c7bd Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 12:17:17 +0200 Subject: [PATCH 11/46] feat(provider-tck): implement the shutdown, metadata and error-message steps Follow the conformance assets to open-feature/spec@15fe861, which adds metadata.feature, three shutdown scenarios to lifecycle.feature, the falsy-value and integer-precision scenarios to evaluation.feature, the lossless half of @numeric-coercion to errors.feature, and six flags to the canonical set. Steps: - "the error message should be empty" reads the last evaluation's error_message and accepts None or "". - "the provider is shut down" and "the provider is initialized again" call the registered provider's own shutdown() and initialize() directly, not through the SDK, so a scenario can shut down twice and an evaluation afterwards reaches the instance that was brought back. Each call is recorded as a LifecycleRecord with its duration and anything it raised; "no exception should have been thrown" now reads those records alongside the evaluation's, so there is one mechanism rather than two. A call that outlasts ready_timeout is given up on and recorded as a TimeoutError. - "the shutdown should have completed within {int}ms" bounds the most recent shutdown, parsed the way the event step's bound is. - "the provider metadata name should not be empty" asks the provider for get_metadata() and requires a non-blank string. Capabilities and flags: - @large-integers is a declarable capability. Python's int is unbounded, so both in-memory self-tests declare it. - The six new flags are transcribed into canonical_flag_set(), and a test checks the transcription against canonical-flags.json value for value and Python type for Python type, so 10.0 stays a float and false, 0 and "" stay values. - The in-memory self-tests stop declaring @numeric-coercion: the SDK's InMemoryProvider hands values back untouched and the client's type check is isinstance-based, so 10.0 requested as an integer is a TYPE_MISMATCH rather than 10. The lossless scenarios exist to catch exactly that, and the capability is optional, so the honest declaration is to leave it out. Recorded as finding 3 in the README. The lifecycle steps have no canonical scenario running them here, because neither in-memory suite declares @lifecycle, so test_lifecycle_steps drives them against a recording provider instead. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 60 +++- tools/openfeature-provider-tck/spec | 2 +- .../contrib/tools/provider_tck/capability.py | 35 ++- .../contrib/tools/provider_tck/config.py | 8 +- .../contrib/tools/provider_tck/provider.py | 35 ++- .../contrib/tools/provider_tck/state.py | 78 ++++- .../tools/provider_tck/steps/flag_steps.py | 49 ++- .../provider_tck/steps/provider_steps.py | 150 +++++++++- .../tests/test_controllable_conformance.py | 8 +- .../tests/test_in_memory_conformance.py | 15 +- .../tests/test_in_process_control.py | 51 ++++ .../tests/test_lifecycle_steps.py | 278 ++++++++++++++++++ 12 files changed, 725 insertions(+), 44 deletions(-) create mode 100644 tools/openfeature-provider-tck/tests/test_lifecycle_steps.py diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index 3669b0e18..d5f48d83c 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -148,7 +148,7 @@ suite at all, so `pytest.skip` carries the reason into the report: ``` SKIPPED provider does not declare capability @stale. - Declared: @events @numeric-coercion @object + Declared: @events @large-integers @object ``` | Capability | Tag | Meaning | @@ -160,6 +160,7 @@ SKIPPED provider does not declare capability @stale. | `Capability.OBJECT` | `@object` | supports structured flag values | | `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | | `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | +| `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `Capability.TARGETING` | `@targeting` | reserved; **not declarable** — no scenarios yet | | `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | @@ -198,10 +199,32 @@ which is scoped to flagd's own implementations, and the tag carries that name borrowed name. **A provider that behaves differently is not violating the specification**, so withholding this capability may be a deliberate choice as readily as a defect. -Only the lossy half is tested. The canonical flag set has no integral float to ask the lossless half -of, so a provider that wrongly rejects `10.0` as an integer still passes; adding one changes the flag -set for every language at once. Appendix F records that as an open gap, together with a second one: -the width of a language's integer accessor — 64-bit against 32-bit — is not modelled at all. +Both halves are tested, and a provider declaring the tag must satisfy all three scenarios: `float-flag` +(`0.5`) requested as an integer is a `TYPE_MISMATCH`; `integral-float-flag` (`10.0`) requested as an +integer is `10`; `integer-flag` (`10`) requested as a float is `10.0`. Rejecting every float is an easy +way to pass the first, and the other two are what stop it. + +The width of the integer accessor is the related property, and it is a capability of its own because +it belongs to the SDK rather than to the provider. Every language can ask for 2^31 − 1, so that +precision scenario is untagged; only the one asking for 2^53 − 1 carries `@large-integers`. A Python +`int` is unbounded, so a Python provider declares it unless something of its own — a 32-bit field in +its wire format, a float on the way through — narrows the value. + +### Steps that reach the provider directly + +Everything the suite asks of a provider goes through an OpenFeature client, as an application's +would — except three steps. `the provider is shut down` and `the provider is initialized again` call +the provider's own `shutdown()` and `initialize()` on the registered instance, and +`the provider metadata name should not be empty` asks it for `get_metadata()`. Going through the SDK +would test the registry's bookkeeping as much as the provider, and Appendix B already does that; it +would also make a double shutdown impossible to express, since the registry calls `shutdown` once +per registration. + +The registry is not told. The client keeps pointing at the same instance, so an evaluation after +re-initialising reaches the very object that was shut down and brought back. When the scenario ends, +the SDK shuts the provider down once more on its own — requirement 2.5.3 makes that second call +harmless, and the suite relies on it. A direct call that outlasts `TckConfig.ready_timeout` is given +up on and fails its scenario with a message rather than hanging the session. ### Declaring more than a capability set @@ -257,7 +280,7 @@ those scenarios are skipped with their reason. ## Findings -Two, both confirmed by running the suite rather than by reading code. +Three, all confirmed by running the suite rather than by reading code. ### 1. A boolean satisfies an Integer request @@ -283,6 +306,18 @@ Only half the machinery is missing — `AbstractProvider` already supplies `emit_provider_configuration_changed` — which is why `ControllableInMemoryProvider` here is a small subclass rather than a reimplementation, and why it should port back to the SDK as a method. +### 3. The in-memory provider does not coerce numbers + +`integral-float-flag` (`10.0`) requested as an integer returns the code default with `TYPE_MISMATCH`, +and `integer-flag` (`10`) requested as a float does the same. The provider hands each variant back +untouched and the client's type check is `isinstance`-based, so neither lossless direction happens. +The lossy scenario passes — every float is rejected — which is exactly the shortcut the two lossless +scenarios exist to catch. + +This is not a defect: `@numeric-coercion` is optional, and the specification does not define the +behaviour. So neither in-memory self-test declares the tag, and the three scenarios are skipped with +that reason rather than failing. + ## Where the assets come from The Gherkin feature files, the canonical flag set and the control-API document are **not owned by @@ -316,21 +351,24 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | --- | --- | --- | | `test_in_memory_conformance` | the SDK's `InMemoryProvider` | reference adoption for a backend-less provider | | `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | -| `test_in_process_control` | `InProcessControl` | pins what the Gherkin cannot assert about itself | +| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set mirrors `canonical-flags.json` type for type | +| `test_lifecycle_steps` | the steps that call the provider directly | the in-memory suites skip `@lifecycle`, so the shutdown, re-initialise and metadata steps are driven against a recording provider instead | | `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | | `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity | ``` -84 passed, 9 skipped, 2 xfailed +106 passed, 21 skipped, 2 xfailed ``` No Docker and no network. The conformance suites take under a second; `test_extensions` takes most of the rest, because the properties it checks are properties of a whole pytest session and it runs a generated adoption in a subprocess to check them. -Neither in-memory suite declares `@lifecycle`, so the three lifecycle scenarios are skipped in both. -That is the point: with no backend to reach, they would pass without testing anything — which is -what they did while the feature was gated on `@events`. +Neither in-memory suite declares `@lifecycle`, so the six lifecycle scenarios — three about +initialisation, three about shutdown — are skipped in both. That is the point: with no backend to +reach, the initialisation ones would pass without testing anything — which is what they did while the +feature was gated on `@events`. Neither declares `@numeric-coercion` either, for the reason in +finding 3, so its three scenarios are skipped too. ## Known gaps diff --git a/tools/openfeature-provider-tck/spec b/tools/openfeature-provider-tck/spec index dc4d7ae8d..15fe86117 160000 --- a/tools/openfeature-provider-tck/spec +++ b/tools/openfeature-provider-tck/spec @@ -1 +1 @@ -Subproject commit dc4d7ae8df1c664f82a4adf46cd43812980c0da3 +Subproject commit 15fe861170f463c20743f5fdb6f7ea083f405f80 diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py index 8be2b2403..b1891daf6 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py @@ -88,12 +88,35 @@ class Capability(str, Enum): that, and flagd's instance is tracked as `open-feature/flagd#1996 `_. - Only the lossy half has a scenario. The canonical flag set has no integral - float to ask the lossless half of, and adding one changes the flag set for - every language at once, so a provider that wrongly rejects ``10.0`` as an - integer still passes. Appendix F records that as an open gap, along with a - second one: the width of a language's integer accessor is not modelled here - at all. + Both halves have scenarios, and a provider declaring the tag must satisfy + all three. The lossy half asks for ``float-flag`` (``0.5``) as an integer + and expects ``TYPE_MISMATCH``; the lossless half asks for + ``integral-float-flag`` (``10.0``) as an integer and for ``integer-flag`` + (``10``) as a float, and expects both to succeed. Rejecting every float is + an easy way to pass the first, and the other two are what stop it. + + The SDK's own ``InMemoryProvider`` cannot declare this: it hands values + back untouched and the client's type check is ``isinstance``-based, so + ``10.0`` requested as an integer is a ``TYPE_MISMATCH`` rather than ``10``. + The width of the integer accessor is a separate property, and a separate + capability: :attr:`LARGE_INTEGERS`. + """ + + LARGE_INTEGERS = "large-integers" + """Provider resolves integers up to 2^53 - 1 exactly. + + A property of the language's SDK as much as of the provider, which is why + it is a capability rather than mandatory: Java's integer accessor is a + 32-bit ``Integer``, and a provider cannot resolve a value the accessor has + no room for. Every language can ask for 2^31 - 1, so that precision + scenario is untagged; only the one asking for 2^53 - 1 carries this tag. + + Python's ``int`` is unbounded, so a Python provider declares it unless + something of its own -- a 32-bit field in its wire format, a float on the + way through -- narrows the value. Nothing above 2^53 - 1 is asked for: + JavaScript cannot represent it, and what a provider owes a value that does + not fit the requested accessor is the open question in + `open-feature/spec#430 `_. """ TARGETING = "targeting" diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py index 384702669..71127051d 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py @@ -179,7 +179,13 @@ class TckConfig: """ ready_timeout: float = DEFAULT_READY_TIMEOUT - """Seconds to wait for a provider to reach ``READY`` during initialisation.""" + """Seconds to wait for a provider to reach ``READY`` during initialisation. + + Also the longest the suite waits on a direct ``shutdown`` or ``initialize`` + call before giving up on it and recording the wait as a failure, so that a + provider whose shutdown hangs on a backend that is gone fails its scenario + with a message rather than hanging the session. + """ def __post_init__(self) -> None: problems: list[str] = [] diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py index 33de67762..cc244b5b6 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py @@ -93,14 +93,22 @@ def changing_flag(default_variant: str) -> InMemoryFlag[str]: def canonical_flag_set() -> FlagStorage: """Return the canonical flag set as SDK in-memory flags. - Mirrors ``flag_data/canonical-flags.json`` entry for entry. Two properties - of that file are load-bearing and hold here too: + Mirrors ``flag_data/canonical-flags.json`` entry for entry -- and the + self-tests check that it does, value for value and Python type for Python + type. Four properties of that file are load-bearing and hold here too: * ``missing-flag`` is absent, which is what the ``FLAG_NOT_FOUND`` scenario tests. Adding it turns that scenario green for the wrong reason. * no flag carries a ``context_evaluator``, so every evaluation reports reason ``STATIC`` -- the TCK tests a provider's mapping of a response, not a backend's evaluation logic. + * ``false-flag``, ``zero-flag`` and ``empty-string-flag`` resolve to + ``False``, ``0`` and ``""``. They are values, not absences, and the falsy + scenarios exist to catch a provider that cannot tell the difference. + * ``integral-float-flag`` is the ``float`` ``10.0`` and ``huge-integer-flag`` + is the ``int`` ``9007199254740991``. Writing the first as ``10`` makes the + lossless-coercion scenario pass without coercing; nothing here goes + through a float, so the second cannot be rounded. """ return { "boolean-flag": InMemoryFlag( @@ -115,6 +123,29 @@ def canonical_flag_set() -> FlagStorage: "float-flag": InMemoryFlag( default_variant="half", variants={"tenth": 0.1, "half": 0.5} ), + # 2^31 - 1: the largest value every language's integer accessor can ask for. + "large-integer-flag": InMemoryFlag( + default_variant="max-int32", variants={"one": 1, "max-int32": 2147483647} + ), + # 2^53 - 1: asked for only under @large-integers. A Python int is exact. + "huge-integer-flag": InMemoryFlag( + default_variant="max-safe", + variants={"one": 1, "max-safe": 9007199254740991}, + ), + # A float with no fractional part, for the lossless half of + # @numeric-coercion. The trailing ``.0`` is the whole point. + "integral-float-flag": InMemoryFlag( + default_variant="ten", variants={"tenth": 0.1, "ten": 10.0} + ), + "false-flag": InMemoryFlag( + default_variant="off", variants={"on": True, "off": False} + ), + "zero-flag": InMemoryFlag( + default_variant="zero", variants={"one": 1, "zero": 0} + ), + "empty-string-flag": InMemoryFlag( + default_variant="empty", variants={"greeting": "hi", "empty": ""} + ), "object-flag": InMemoryFlag( default_variant="template", variants={ diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py index 71ea41508..074005f59 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py @@ -15,10 +15,11 @@ from openfeature.client import OpenFeatureClient from openfeature.event import EventDetails, ProviderEvent from openfeature.flag_evaluation import FlagType +from openfeature.provider import FeatureProvider from .config import TckConfig -__all__ = ["EvaluationRecord", "EventRecorder", "TckState"] +__all__ = ["EvaluationRecord", "EventRecorder", "LifecycleRecord", "TckState"] @dataclass @@ -39,6 +40,29 @@ class EvaluationRecord: """ +@dataclass +class LifecycleRecord: + """The outcome of one direct call into the provider's lifecycle. + + The shutdown scenarios call the provider's own ``shutdown`` and + ``initialize`` rather than going through the SDK, because the SDK's + bookkeeping around them is Appendix B's business rather than this suite's. + Each call is recorded the same way an evaluation is -- what it raised, if + anything -- so that "no exception should have been thrown" reads one kind + of record for both, plus how long it took, which is what the prompt-shutdown + scenario bounds. + """ + + operation: str + """``shutdown`` or ``initialize``, for failure messages.""" + + duration: float + """Wall-clock seconds the call took to return, or to be given up on.""" + + raised: BaseException | None = None + """The exception the call raised, if any.""" + + class EventRecorder: """Captures the events of one type, in order, so a scenario consumes them one at a time. @@ -93,10 +117,21 @@ class TckState: config: TckConfig client: OpenFeatureClient | None = None + provider: FeatureProvider | None = None + """The provider under test, for the steps that call it directly. + + Everything else reaches the provider through :attr:`client`, which is how + an application would. The lifecycle and metadata steps are the exception: + they ask the provider itself, because what they verify is the provider's + own ``shutdown``, ``initialize`` and ``get_metadata`` rather than the SDK's + handling of them. + """ flag_key: str | None = None flag_type: FlagType | None = None default_value: typing.Any = None last: EvaluationRecord | None = None + lifecycle: list[LifecycleRecord] = field(default_factory=list) + """Every direct lifecycle call this scenario made, in order.""" remembered: typing.Any = None has_memory: bool = False recorders: dict[ProviderEvent, EventRecorder] = field(default_factory=dict) @@ -111,6 +146,47 @@ def require_client(self) -> OpenFeatureClient: raise AssertionError(msg) return self.client + def require_provider(self) -> FeatureProvider: + if self.provider is None: + msg = ( + "no provider has been registered in this scenario: a " + '"Given a stable provider" or "Given a unavailable provider" step ' + "must come first" + ) + raise AssertionError(msg) + return self.provider + + def require_shutdown(self) -> LifecycleRecord: + """The most recent direct ``shutdown`` call, for the steps that bound it.""" + for record in reversed(self.lifecycle): + if record.operation == "shutdown": + return record + msg = ( + "the provider has not been shut down in this scenario: a " + '"When the provider is shut down" step must come first' + ) + raise AssertionError(msg) + + def raised(self) -> list[tuple[str, BaseException]]: + """Every call into the provider that raised, as (what was called, exception). + + The evaluation and the lifecycle calls are recorded separately, since + they carry different things, but "did anything the scenario asked of + the provider raise" is one question and this is where it is answered. + """ + raised: list[tuple[str, BaseException]] = [ + (record.operation, record.raised) + for record in self.lifecycle + if record.raised is not None + ] + if self.last is not None and self.last.raised is not None: + raised.append(("the evaluation", self.last.raised)) + return raised + + def has_called_provider(self) -> bool: + """Whether the scenario has asked anything of the provider yet.""" + return self.last is not None or bool(self.lifecycle) + def require_flag(self) -> tuple[str, FlagType, typing.Any]: if self.flag_key is None or self.flag_type is None: msg = ( diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py index f284eb5b9..812c5047b 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py @@ -16,6 +16,7 @@ "a_flag_with_key_and_default", "no_exception_should_have_been_thrown", "the_error_code_should_be", + "the_error_message_should_be_empty", "the_flag_was_evaluated_with_details", "the_flag_was_modified", "the_reason_should_be", @@ -147,23 +148,57 @@ def the_error_code_should_be(tck_state: TckState, expected: str) -> None: raise AssertionError(msg) +@then("the error message should be empty") +def the_error_message_should_be_empty(tck_state: TckState) -> None: + """Assert no error message was reported (requirement 2.3.2). + + Asserted on the success paths, where a message contradicts the value beside + it: an application reading the message will believe the wrong one of the + two signals. ``None`` and ``""`` are both "none": the SDK's resolution + details default the field to ``None`` and a provider that writes the empty + string has said the same thing. + """ + record = tck_state.require_evaluation() + if record.error_message: + msg = ( + f"an error message was reported alongside a successful evaluation: " + f"{record.error_message!r}. A value and an error message are two " + f"contradictory signals, and the application cannot tell which to believe" + ) + raise AssertionError(msg) + + @then("no exception should have been thrown") def no_exception_should_have_been_thrown(tck_state: TckState) -> None: - """Assert the evaluation returned rather than raised. + """Assert that nothing the scenario asked of the provider raised. + + That is the evaluation, if there was one, and every direct lifecycle call: + each records what it raised rather than propagating it, and this is the + one step that reads those records back. In Python an errored evaluation returns the code default in the details and does not raise, so this holds on the error paths too. A provider that raises - instead takes the calling application down with it, which is what the - feature files forbid. + instead takes the calling application down with it -- and one that raises + from ``shutdown`` does so from the application's own shutdown, where an + exception is least welcome. Both are what the feature files forbid. """ - record = tck_state.require_evaluation() - if record.raised is not None: + if not tck_state.has_called_provider(): msg = ( - f"the evaluation raised {record.raised!r}. A flag evaluation must always " - f"return a value and an error code, never raise" + "nothing has been asked of the provider in this scenario: a " + '"When the flag was evaluated with details" or "When the provider is ' + 'shut down" step must come first' ) raise AssertionError(msg) + raised = tck_state.raised() + if raised: + what, exc = raised[0] + msg = ( + f"{what} raised {exc!r}. A provider must return from an evaluation with a " + f"value and an error code, and from a lifecycle call quietly -- never raise" + ) + raise AssertionError(msg) from exc + @then("the resolved object value should contain") def the_resolved_object_value_should_contain( diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py index bca37faee..453557e64 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py @@ -1,18 +1,27 @@ -"""Steps that put a provider under test.""" +"""Steps that put a provider under test, and the ones that talk to it directly.""" from __future__ import annotations import concurrent.futures import contextlib +import time +from collections.abc import Callable -from pytest_bdd import given, parsers +from pytest_bdd import given, parsers, then, when from openfeature import api -from openfeature.provider import FeatureProvider +from openfeature.evaluation_context import EvaluationContext -from ..state import TckState +from ..state import LifecycleRecord, TckState -__all__ = ["a_stable_provider", "an_unavailable_provider"] +__all__ = [ + "a_stable_provider", + "an_unavailable_provider", + "the_provider_is_initialized_again", + "the_provider_is_shut_down", + "the_provider_metadata_name_should_not_be_empty", + "the_shutdown_should_have_completed_within", +] @given(parsers.re(r"^an? stable provider$")) @@ -32,7 +41,9 @@ def a_stable_provider(tck_state: TckState) -> None: raise AssertionError(msg) try: - _set_provider_within(provider, config.domain, config.ready_timeout) + _call_within( + lambda: api.set_provider(provider, config.domain), config.ready_timeout + ) except TimeoutError: msg = ( f"the provider did not become ready within {config.ready_timeout}s. The backend " @@ -48,6 +59,7 @@ def a_stable_provider(tck_state: TckState) -> None: ) raise AssertionError(msg) from exc + tck_state.provider = provider tck_state.client = api.get_client(config.domain) @@ -87,24 +99,136 @@ def an_unavailable_provider(tck_state: TckState) -> None: with contextlib.suppress(Exception): api.set_provider(provider, config.domain) + tck_state.provider = provider tck_state.client = api.get_client(config.domain) -def _set_provider_within( - provider: FeatureProvider, domain: str, timeout: float +@when("the provider is shut down") +def the_provider_is_shut_down(tck_state: TckState) -> None: + """Call the provider's own ``shutdown``, directly. + + Not through the SDK. The SDK shuts a provider down when it is replaced or + when the API is shut down, but going that way would test the registry's + bookkeeping as much as the provider, and Appendix B already does that. + Calling ``shutdown`` on the instance is also what lets a scenario call it + twice: the registry only ever calls it once per registration. + + The registry is not told. The client still points at the same instance, so + an evaluation after "the provider is initialized again" reaches the very + object that was shut down and brought back, which is what that scenario + asserts. And when the scenario ends the SDK shuts the provider down once + more on its own -- a second call, which requirement 2.5.3 makes harmless. + """ + _record_lifecycle_call(tck_state, "shutdown", tck_state.require_provider().shutdown) + + +@when("the provider is initialized again") +def the_provider_is_initialized_again(tck_state: TckState) -> None: + """Call the provider's own ``initialize`` after it was shut down. + + With an empty context, as the SDK would with none set. Direct for the same + reason as the shutdown step: re-registering through the SDK would create a + new registration around the same instance, and what is under test is that + the instance itself reverts to an initialisable state. + """ + provider = tck_state.require_provider() + _record_lifecycle_call( + tck_state, "initialize", lambda: provider.initialize(EvaluationContext()) + ) + + +@then(parsers.re(r"^the shutdown should have completed within (?P\d+)ms$")) +def the_shutdown_should_have_completed_within(tck_state: TckState, millis: str) -> None: + """Bound the most recent shutdown. + + The scenario using this runs against a backend that will never answer, so + what it asserts is that shutdown returns rather than waiting for a graceful + close that cannot happen. A shutdown that was given up on because it + outlasted ``TckConfig.ready_timeout`` fails here too: its recorded duration + is however long the suite waited before moving on. + """ + record = tck_state.require_shutdown() + bound = int(millis) / 1000.0 + if record.duration > bound: + msg = ( + f"shutdown took {record.duration * 1000:.0f}ms, expected it to complete within " + f"{millis}ms. A shutdown that waits on a backend that is gone hangs the host " + f"application's own shutdown" + ) + raise AssertionError(msg) + + +@then("the provider metadata name should not be empty") +def the_provider_metadata_name_should_not_be_empty(tck_state: TckState) -> None: + """Assert the provider identifies itself (requirement 2.1.1). + + Asked of the provider rather than of ``api.get_provider_metadata``, which + would answer for whatever the registry holds under the domain: the same + object here, but the question is about the provider. + """ + provider = tck_state.require_provider() + try: + metadata = provider.get_metadata() + except Exception as exc: + msg = f"get_metadata raised {exc!r}: the provider cannot say what it is" + raise AssertionError(msg) from exc + + name = getattr(metadata, "name", None) + if not isinstance(name, str) or not name.strip(): + msg = ( + f"the provider metadata name is {name!r}, expected a non-empty string. A " + f"conformance report keyed on the name cannot be attributed without one" + ) + raise AssertionError(msg) + + +def _record_lifecycle_call( + tck_state: TckState, operation: str, call: Callable[[], object] ) -> None: - """Register a provider, giving up if initialisation has not returned in time. + """Make one direct lifecycle call and record how it went, raising nothing. + + An exception is recorded rather than propagated, for the same reason an + evaluation's is: "no exception should have been thrown" is a step of its + own, and a scenario that wants a raise to fail says so there. Only + ``Exception`` is caught, though. The shutdown scenario that matters most is + the one against a backend that is gone, which is exactly where somebody + might reach for Ctrl-C, and a ``KeyboardInterrupt`` recorded as "shutdown + raised" would carry the run on past the thing they interrupted. + + A call that outlasts ``TckConfig.ready_timeout`` is given up on and recorded + as a ``TimeoutError`` with the time waited, so a hanging shutdown fails its + scenario with a message rather than hanging the session. + """ + started = time.perf_counter() + raised: BaseException | None = None + try: + _call_within(call, tck_state.config.ready_timeout) + except TimeoutError: + raised = TimeoutError( + f"{operation} did not return within {tck_state.config.ready_timeout}s" + ) + except Exception as exc: # recorded here, asserted on by its own step + raised = exc + duration = time.perf_counter() - started + tck_state.lifecycle.append( + LifecycleRecord(operation=operation, duration=duration, raised=raised) + ) + + +def _call_within(call: Callable[[], object], timeout: float) -> None: + """Make a call into the provider, giving up if it has not returned in time. - ``api.set_provider`` initialises synchronously and has no timeout of its own, so a - provider that hangs while connecting would hang the whole session with no useful - message. Running it on a worker thread bounds it. + Neither ``api.set_provider`` nor a provider's own ``shutdown`` has a timeout + of its own, so one that hangs while talking to its backend would hang the + whole session with no useful message. Running it on a worker thread bounds + it. The worker is deliberately not cancelled on timeout -- Python cannot interrupt a thread blocked in a socket call -- so it is left to finish or die with the process. That is acceptable here because a timeout already means the scenario is failing. """ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: - future = pool.submit(api.set_provider, provider, domain) + future = pool.submit(call) try: future.result(timeout=timeout) except concurrent.futures.TimeoutError: diff --git a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py index 77b3c3d0b..1d36dec21 100644 --- a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py @@ -38,6 +38,12 @@ def tck_config() -> TckConfig: ``test_in_memory_conformance``: there is no backend to reach during initialisation, so the readiness scenario would pass here without testing anything. It did exactly that while the feature was gated on ``@events``. + + ``NUMERIC_COERCION`` stays undeclared for the reason given there too. + ``ControllableInMemoryProvider`` changes nothing about resolution, so it + inherits the SDK provider's refusal to coerce: ``10.0`` requested as an + integer is a ``TYPE_MISMATCH`` rather than ``10``. ``LARGE_INTEGERS`` is + declared, since a Python ``int`` is exact at 2^53 - 1. """ control = InProcessControl() return TckConfig( @@ -48,7 +54,7 @@ def tck_config() -> TckConfig: Capability.EVENTS, Capability.CONFIGURATION_CHANGE, Capability.OBJECT, - Capability.NUMERIC_COERCION, + Capability.LARGE_INTEGERS, }, ) diff --git a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py index 9b9e4a191..18fef5c05 100644 --- a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py @@ -93,6 +93,19 @@ def tck_config() -> TckConfig: vacuously while the feature was gated on ``EVENTS``, which is precisely the failure mode the split of ``@lifecycle`` from ``@events`` exists to end. A skip with a reason is the honest outcome. + * ``NUMERIC_COERCION`` -- omitted because the SDK's in-memory provider does + not coerce. It hands each variant back untouched, and the client's type + check is ``isinstance``-based, so ``integral-float-flag`` (``10.0``) + requested as an integer is a ``TYPE_MISMATCH`` rather than ``10``, and + ``integer-flag`` (``10``) requested as a float is one rather than + ``10.0``. The lossy scenario passes for the wrong reason -- every float + is rejected -- which is exactly what the two lossless scenarios exist to + catch, and declaring the tag would have them catch it here. The + capability is optional, so this is a choice the provider is entitled to + rather than a deviation. + + ``LARGE_INTEGERS`` is declared: a Python ``int`` is unbounded and nothing + in this provider routes a value through a float. """ return TckConfig( name="in-memory", @@ -101,7 +114,7 @@ def tck_config() -> TckConfig: capabilities={ Capability.EVENTS, Capability.OBJECT, - Capability.NUMERIC_COERCION, + Capability.LARGE_INTEGERS, }, ) diff --git a/tools/openfeature-provider-tck/tests/test_in_process_control.py b/tools/openfeature-provider-tck/tests/test_in_process_control.py index 88322acfd..ea19864ef 100644 --- a/tools/openfeature-provider-tck/tests/test_in_process_control.py +++ b/tools/openfeature-provider-tck/tests/test_in_process_control.py @@ -6,6 +6,9 @@ from __future__ import annotations +import json +import typing + import pytest from openfeature.contrib.tools.provider_tck import ( @@ -14,6 +17,7 @@ ControllableInMemoryProvider, InProcessControl, canonical_flag_set, + canonical_flags_json, ) from openfeature.event import ProviderEvent @@ -121,6 +125,53 @@ def test_canonical_flag_set_omits_missing_flag() -> None: assert "missing-flag" not in canonical_flag_set() +def _same_value_and_type(expected: typing.Any, actual: typing.Any) -> bool: + """Equal, and of the same Python type, member by member. + + ``==`` alone is what a seeding step that "cleans up" gets past: ``10 == 10.0`` + and ``0 == False`` in Python, so the integral float and the falsy values + would compare equal to exactly the mistranslations they exist to catch. + """ + if type(expected) is not type(actual): + return False + if isinstance(expected, dict): + return set(expected) == set(actual) and all( + _same_value_and_type(v, actual[k]) for k, v in expected.items() + ) + if isinstance(expected, list): + return len(expected) == len(actual) and all( + _same_value_and_type(e, a) for e, a in zip(expected, actual, strict=True) + ) + return bool(expected == actual) + + +def test_canonical_flag_set_mirrors_the_canonical_json_type_for_type() -> None: + """The in-memory flag set is transcribed, so this is what stops it drifting. + + Key for key, default variant for default variant, and every variant's value + with its Python type: ``json.loads`` keeps ``10.0`` a ``float`` and ``0`` + an ``int``, and the transcription has to as well. The four load-bearing + properties the flag file documents -- no ``missing-flag``, no targeting, + falsy values kept, ``10.0`` a float and 2^53 - 1 an integer -- all follow + from being an exact mirror of it. + """ + canonical = json.loads(canonical_flags_json())["flags"] + transcribed = canonical_flag_set() + + assert set(transcribed) == set(canonical) + for key, definition in canonical.items(): + flag = transcribed[key] + assert flag.default_variant == definition["defaultVariant"], key + assert flag.context_evaluator is None, f"{key} has targeting" + assert set(flag.variants) == set(definition["variants"]), key + for variant, value in definition["variants"].items(): + assert _same_value_and_type(value, flag.variants[variant]), ( + f"{key}/{variant}: canonical {value!r} ({type(value).__name__}), " + f"transcribed {flag.variants[variant]!r} " + f"({type(flag.variants[variant]).__name__})" + ) + + def test_update_flags_names_the_union_of_old_and_new_keys() -> None: """Appendix A asks for the union, not just the new keys. diff --git a/tools/openfeature-provider-tck/tests/test_lifecycle_steps.py b/tools/openfeature-provider-tck/tests/test_lifecycle_steps.py new file mode 100644 index 000000000..9f2024395 --- /dev/null +++ b/tools/openfeature-provider-tck/tests/test_lifecycle_steps.py @@ -0,0 +1,278 @@ +"""The steps that talk to the provider directly, pinned outside the Gherkin. + +The shutdown scenarios live in ``lifecycle.feature``, which is gated on +``@lifecycle``, and neither in-memory self-test declares that -- there is no +backend to reach, so the readiness scenario would pass without testing anything. +That leaves the shutdown, re-initialise and shutdown-bound steps with no +canonical scenario running them here, and a step that first runs in a +containerised adopter's suite fails there looking like a provider defect. + +So they are driven directly, with a provider that records what was called of it +and can be told to misbehave. What is pinned is the contract the feature file +relies on: that the calls reach the provider's *own* methods rather than the +SDK's, that a raise is recorded and surfaces through the one "no exception" +step rather than through a second mechanism, that the client still reaches the +instance after it was brought back, and that the scenario's teardown copes with +a provider that was shut down underneath it. +""" + +from __future__ import annotations + +import typing +from collections.abc import Iterator + +import pytest + +from openfeature import api +from openfeature.contrib.tools.provider_tck import ( + Capability, + TckConfig, + TckState, + canonical_flag_set, +) +from openfeature.contrib.tools.provider_tck.steps.flag_steps import ( + a_flag_with_key_and_default, + no_exception_should_have_been_thrown, + the_flag_was_evaluated_with_details, + the_resolved_value_should_be, +) +from openfeature.contrib.tools.provider_tck.steps.provider_steps import ( + a_stable_provider, + the_provider_is_initialized_again, + the_provider_is_shut_down, + the_provider_metadata_name_should_not_be_empty, + the_shutdown_should_have_completed_within, +) +from openfeature.evaluation_context import EvaluationContext +from openfeature.provider import Metadata +from openfeature.provider.in_memory_provider import InMemoryProvider + + +class RecordingProvider(InMemoryProvider): + """The SDK's in-memory provider, remembering its lifecycle calls. + + ``fail_shutdown`` and ``fail_initialize`` make the corresponding call raise, + which is how the recording half of the steps is checked; ``metadata_name`` + is what the metadata step is checked against. + """ + + def __init__(self) -> None: + super().__init__(canonical_flag_set()) + self.calls: list[str] = [] + self.fail_shutdown = False + self.fail_initialize = False + self.metadata_name: typing.Any = "recording" + + def initialize(self, evaluation_context: EvaluationContext) -> None: + self.calls.append("initialize") + if self.fail_initialize: + msg = "initialize refused" + raise RuntimeError(msg) + + def shutdown(self) -> None: + self.calls.append("shutdown") + if self.fail_shutdown: + msg = "already closed" + raise RuntimeError(msg) + + def get_metadata(self) -> Metadata: + return Metadata(name=self.metadata_name) + + +class _NoControl: + @property + def description(self) -> str: + return "nothing" + + def prepare_scenario(self) -> None: ... + + def change_flag(self) -> None: ... + + +@pytest.fixture +def provider() -> RecordingProvider: + return RecordingProvider() + + +@pytest.fixture +def state(provider: RecordingProvider) -> Iterator[TckState]: + """A scenario's state, with the recording provider registered by the real step. + + Through ``a_stable_provider`` rather than by hand, so what is tested is the + hand-off the feature files rely on: the step that registers the provider is + the one that makes it available to the steps that call it directly. + """ + config = TckConfig( + name="lifecycle-steps", + control=_NoControl(), + new_provider=lambda: provider, + capabilities={Capability.EVENTS}, + ) + state = TckState(config=config) + a_stable_provider(state) + yield state + state.teardown() + api.shutdown() + api.clear_providers() + + +# -- the calls reach the provider itself ------------------------------------- + + +def test_shutdown_calls_the_providers_own_shutdown_each_time( + state: TckState, provider: RecordingProvider +) -> None: + """Twice asked, twice called -- which the SDK would never do on its own. + + The registry shuts a provider down once per registration. The double-close + scenario needs two calls on one instance, and gets them only because the + step bypasses the registry. + """ + before = list(provider.calls) + the_provider_is_shut_down(state) + the_provider_is_shut_down(state) + assert provider.calls[len(before) :] == ["shutdown", "shutdown"] + assert [record.operation for record in state.lifecycle] == ["shutdown", "shutdown"] + no_exception_should_have_been_thrown(state) + + +def test_initialize_again_reaches_the_same_instance_the_client_uses( + state: TckState, provider: RecordingProvider +) -> None: + """The scenario's whole point: after the round trip, the client serves flags + from the very object that was shut down and brought back. + """ + the_provider_is_shut_down(state) + the_provider_is_initialized_again(state) + assert provider.calls[-2:] == ["shutdown", "initialize"] + + a_flag_with_key_and_default(state, "Boolean", "boolean-flag", "false") + the_flag_was_evaluated_with_details(state) + the_resolved_value_should_be(state, "true") + no_exception_should_have_been_thrown(state) + assert state.client is not None + assert state.client.get_provider_status().value == "READY" + + +def test_initialize_again_passes_an_empty_context( + state: TckState, provider: RecordingProvider +) -> None: + seen: list[EvaluationContext] = [] + original = provider.initialize + + def spy(evaluation_context: EvaluationContext) -> None: + seen.append(evaluation_context) + original(evaluation_context) + + provider.initialize = spy # type: ignore[method-assign] + the_provider_is_initialized_again(state) + assert len(seen) == 1 + assert seen[0].attributes == {} + assert seen[0].targeting_key is None + + +# -- a raise is recorded, and surfaces through the one step ------------------ + + +def test_a_raising_shutdown_fails_the_no_exception_step( + state: TckState, provider: RecordingProvider +) -> None: + """Recorded, not propagated: the step returns and the assertion is elsewhere.""" + provider.fail_shutdown = True + the_provider_is_shut_down(state) + + assert state.lifecycle[-1].raised is not None + with pytest.raises(AssertionError, match="shutdown raised RuntimeError"): + no_exception_should_have_been_thrown(state) + + +def test_a_raising_initialize_fails_the_no_exception_step( + state: TckState, provider: RecordingProvider +) -> None: + provider.fail_initialize = True + the_provider_is_shut_down(state) + the_provider_is_initialized_again(state) + + with pytest.raises(AssertionError, match="initialize raised RuntimeError"): + no_exception_should_have_been_thrown(state) + + +def test_a_lifecycle_raise_is_reported_even_after_a_clean_evaluation( + state: TckState, provider: RecordingProvider +) -> None: + """One mechanism for both kinds of call. + + The re-initialise scenario ends with an evaluation and then the no-exception + step. A raise from the shutdown before it must not be hidden behind the + evaluation that went fine. + """ + provider.fail_shutdown = True + the_provider_is_shut_down(state) + provider.fail_initialize = False + the_provider_is_initialized_again(state) + a_flag_with_key_and_default(state, "Boolean", "boolean-flag", "false") + the_flag_was_evaluated_with_details(state) + + assert state.last is not None and state.last.raised is None + with pytest.raises(AssertionError, match="shutdown raised"): + no_exception_should_have_been_thrown(state) + + +def test_the_no_exception_step_needs_something_to_have_been_called( + state: TckState, +) -> None: + """Before anything was asked of the provider the step has nothing to assert, + and says so rather than passing on an empty record.""" + with pytest.raises(AssertionError, match="nothing has been asked of the provider"): + no_exception_should_have_been_thrown(state) + + +# -- the shutdown bound ------------------------------------------------------ + + +def test_the_shutdown_bound_reads_the_most_recent_shutdown(state: TckState) -> None: + the_provider_is_shut_down(state) + the_shutdown_should_have_completed_within(state, "10000") + + state.lifecycle[-1].duration = 11.0 + with pytest.raises(AssertionError, match="shutdown took 11000ms"): + the_shutdown_should_have_completed_within(state, "10000") + + +def test_the_shutdown_bound_needs_a_shutdown(state: TckState) -> None: + with pytest.raises(AssertionError, match="has not been shut down"): + the_shutdown_should_have_completed_within(state, "10000") + + +# -- metadata ---------------------------------------------------------------- + + +def test_the_metadata_step_accepts_a_name_and_refuses_an_empty_one( + state: TckState, provider: RecordingProvider +) -> None: + the_provider_metadata_name_should_not_be_empty(state) + + for empty in ("", " ", None): + provider.metadata_name = empty + with pytest.raises(AssertionError, match="expected a non-empty string"): + the_provider_metadata_name_should_not_be_empty(state) + + +# -- the scenario after this one --------------------------------------------- + + +def test_a_shut_down_provider_does_not_break_the_next_registration( + state: TckState, provider: RecordingProvider +) -> None: + """What the fixture teardown and the next "Given a stable provider" do. + + Both shut the provider down again through the SDK. Requirement 2.5.3 makes + the second call harmless, and the suite relies on that: a provider that was + shut down directly is still the registered one when the scenario ends. + """ + the_provider_is_shut_down(state) + + replacement = RecordingProvider() + api.set_provider(replacement, state.config.domain) + assert state.client is not None + assert state.client.get_boolean_details("boolean-flag", False).value is True From 30a1471f1071fea0bd3e25b92eb30b406eb144c6 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 13:38:20 +0200 Subject: [PATCH 12/46] fix(provider-tck): name the falsy flags what the SDK suite already names them Moves the spec submodule to ba002ce8, which renames the canonical set's three falsy flags -- false-flag, zero-flag and empty-string-flag become boolean-zero-flag, integer-zero-flag and string-zero-flag -- and follows the rename through the in-process control's flag set. The names the TCK invented were its own. Appendix B's SDK suite already had names for these three, flagd-testbed serves that vocabulary, and a provider suite that asks for a different one gets FLAG_NOT_FOUND four times over for no reason other than the disagreement. The three entries are now byte-identical to specification/assets/gherkin/test-flags.json on spec main, so a backend seeded for the SDK suite is already seeded for this one. The variant names move with the keys, from on/off, one/zero and greeting/empty to zero/non-zero throughout. That is not cosmetic: the falsy scenarios assert the variant as well as the value, so a fixture that kept the old variant names would fail on the assertion rather than the lookup. canonical_flag_set() is a transcription of the asset, so it has to move in the same commit: the type-for-type mirror test compares the two, and a commit that moved only the pin would leave the in-process fixture answering FLAG_NOT_FOUND to every falsy scenario -- the same failure the rename exists to remove, in the opposite direction. Nothing generated needed committing. Both the copied assets and spec_revision.json are gitignored and rebuilt by hatch_build_sync.py, which keeps the submodule pin the single record of the revision this package targets. The scenario count is unchanged at 40, as a pure rename should leave it. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/spec | 2 +- .../contrib/tools/provider_tck/provider.py | 20 ++++++++++--------- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/tools/openfeature-provider-tck/spec b/tools/openfeature-provider-tck/spec index 15fe86117..ba002ce8e 160000 --- a/tools/openfeature-provider-tck/spec +++ b/tools/openfeature-provider-tck/spec @@ -1 +1 @@ -Subproject commit 15fe861170f463c20743f5fdb6f7ea083f405f80 +Subproject commit ba002ce8e807ca97920a5ebd8b9303a556f15d29 diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py index cc244b5b6..fae7b5640 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py @@ -102,9 +102,11 @@ def canonical_flag_set() -> FlagStorage: * no flag carries a ``context_evaluator``, so every evaluation reports reason ``STATIC`` -- the TCK tests a provider's mapping of a response, not a backend's evaluation logic. - * ``false-flag``, ``zero-flag`` and ``empty-string-flag`` resolve to - ``False``, ``0`` and ``""``. They are values, not absences, and the falsy - scenarios exist to catch a provider that cannot tell the difference. + * ``boolean-zero-flag``, ``integer-zero-flag`` and ``string-zero-flag`` + resolve to ``False``, ``0`` and ``""``. They are values, not absences, and + the falsy scenarios exist to catch a provider that cannot tell the + difference. Their ``zero``/``non-zero`` variant names are load-bearing + too: the scenarios assert the variant, not only the value. * ``integral-float-flag`` is the ``float`` ``10.0`` and ``huge-integer-flag`` is the ``int`` ``9007199254740991``. Writing the first as ``10`` makes the lossless-coercion scenario pass without coercing; nothing here goes @@ -137,14 +139,14 @@ def canonical_flag_set() -> FlagStorage: "integral-float-flag": InMemoryFlag( default_variant="ten", variants={"tenth": 0.1, "ten": 10.0} ), - "false-flag": InMemoryFlag( - default_variant="off", variants={"on": True, "off": False} + "boolean-zero-flag": InMemoryFlag( + default_variant="zero", variants={"zero": False, "non-zero": True} ), - "zero-flag": InMemoryFlag( - default_variant="zero", variants={"one": 1, "zero": 0} + "integer-zero-flag": InMemoryFlag( + default_variant="zero", variants={"zero": 0, "non-zero": 1} ), - "empty-string-flag": InMemoryFlag( - default_variant="empty", variants={"greeting": "hi", "empty": ""} + "string-zero-flag": InMemoryFlag( + default_variant="zero", variants={"zero": "", "non-zero": "str"} ), "object-flag": InMemoryFlag( default_variant="template", From 3a60c7f458814a60a1c76597ae5714d90037c48c Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 13:52:09 +0200 Subject: [PATCH 13/46] fix(provider-tck): wait for the provider to initialise before evaluating ``a stable provider`` and ``an unavailable provider`` both registered through ``api.set_provider``, which initialises on a worker thread and returns immediately. Both steps needed ``api.set_provider_and_wait``, which is the variant that passes ``wait_for_init=True`` down to the registry. The stable case is the damaging one. The step's docstring already claimed that registration "initialises synchronously and dispatches PROVIDER_READY, so by the time this step returns the provider is ready" -- the claim the whole suite rests on, and it was not true of the call being made. Every scenario therefore ran its first evaluation against a provider still coming up and got ``PROVIDER_NOT_READY``, which looks precisely like a provider that cannot resolve anything. Against a real flagd backend that is 58 of 80 tests failing for a reason that has nothing to do with flagd. The unavailable case was wrong in the mirror image. Its comment reasons that "the SDK's registry already converts a raising initialize into PROVIDER_ERROR", which only happens if ``initialize`` is actually called; with the non-waiting variant registration returned before the provider had tried to reach its backend, so the ``@unavailable`` scenarios asserted an error state that had not happened yet. The ``contextlib.suppress`` around it stays: with the waiting variant a raising ``initialize`` can propagate, and that must not take down a scenario whose subject is the observable error state rather than how registration returned. Nothing in this package's own tests could catch it. Both self-hosted suites drive ``InMemoryProvider``, which initialises in microseconds, so the race was always won and the step's assumption held by accident. It took a backend that takes a moment to come up -- flagd behind a container -- to show the difference, which is also why it survived until the conformance suite could be run for real. Signed-off-by: Simon Schrottner --- .../provider_tck/steps/provider_steps.py | 35 +++++++++++++------ 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py index 453557e64..cf84f8eac 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py @@ -28,11 +28,17 @@ def a_stable_provider(tck_state: TckState) -> None: """Register the provider under test against the running, seeded backend. - ``api.set_provider`` initialises synchronously and dispatches - ``PROVIDER_READY``, so by the time this step returns the provider is ready - and every scenario that follows can assume it. A suite that started + ``api.set_provider_and_wait`` initialises the provider before it returns and + dispatches ``PROVIDER_READY``, so by the time this step returns the provider + is ready and every scenario that follows can assume it. A suite that started evaluating before that would report races in the TCK as defects in the provider. + + It has to be the waiting variant. Plain ``api.set_provider`` registers and + initialises on a worker thread, returning long before the provider is up, so + the very first evaluation of every scenario answers ``PROVIDER_NOT_READY`` -- + a TCK defect that reads exactly like a provider that cannot resolve + anything. """ config = tck_state.config provider = config.new_provider() @@ -42,7 +48,8 @@ def a_stable_provider(tck_state: TckState) -> None: try: _call_within( - lambda: api.set_provider(provider, config.domain), config.ready_timeout + lambda: api.set_provider_and_wait(provider, config.domain), + config.ready_timeout, ) except TimeoutError: msg = ( @@ -92,12 +99,18 @@ def an_unavailable_provider(tck_state: TckState) -> None: msg = "TckConfig.new_unavailable_provider returned None" raise AssertionError(msg) + # The waiting variant for the same reason as the stable provider: plain + # set_provider initialises on a worker thread, so registration would return + # before the provider had even tried to reach its backend, and the scenario + # would assert an error state that had not happened yet. + # # A raising initialize is already converted to PROVIDER_ERROR by the SDK's - # registry, so this is belt and braces: a provider that raises anyway must - # not take the scenario down with it, because the contract is about the - # observable error state rather than about how registration returned. + # registry, so the suppression is belt and braces: a provider that raises + # anyway must not take the scenario down with it, because the contract is + # about the observable error state rather than about how registration + # returned. with contextlib.suppress(Exception): - api.set_provider(provider, config.domain) + api.set_provider_and_wait(provider, config.domain) tck_state.provider = provider tck_state.client = api.get_client(config.domain) @@ -218,9 +231,9 @@ def _record_lifecycle_call( def _call_within(call: Callable[[], object], timeout: float) -> None: """Make a call into the provider, giving up if it has not returned in time. - Neither ``api.set_provider`` nor a provider's own ``shutdown`` has a timeout - of its own, so one that hangs while talking to its backend would hang the - whole session with no useful message. Running it on a worker thread bounds + Neither ``api.set_provider_and_wait`` nor a provider's own ``shutdown`` has a + timeout of its own, so one that hangs while talking to its backend would hang + the whole session with no useful message. Running it on a worker thread bounds it. The worker is deliberately not cancelled on timeout -- Python cannot interrupt a From 042c32925dce46bf3026a527443e383c9a091f75 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 16:06:28 +0200 Subject: [PATCH 14/46] feat(provider-tck): add the @reinitialization capability Bumps the spec pin to fc99d5ac, which gates the scenario "A provider that was shut down can be initialized again" behind a new @reinitialization tag, and adds that capability to the vocabulary. Requirement 2.5.2 says a provider SHOULD revert to its uninitialized state after shutdown, and its supporting text adds that "some providers MAY allow reinitialization from this state". Reuse is permitted, not required, so asserting it unconditionally reported a permitted choice as a conformance failure -- the mirror image of a vacuous pass, and on its way to being filed as a defect against an implementation that was exercising a choice the specification offers it. The capability is declarable without further work: DECLARABLE_CAPABILITIES is the enum minus the reserved set, so it picks the new member up, and the declaration guard in test_declaration.py confirms the coupling -- running the new enum against the old pin fails it, because no scenario there carries the tag. Neither in-repo adoption declares it, and deliberately so. The scenario lives in lifecycle.feature, which carries @lifecycle at the feature level, so it inherits that tag and carries both; the gate skips a scenario when any capability gating it is undeclared. Neither in-memory adoption declares LIFECYCLE -- there is no backend to reach during initialisation -- so the scenario was skipped at the previous pin too, and declaring reuse would be claiming a property nothing has observed. The skip breakdown shows the move exactly: @lifecycle went from 6 skips to 4, with 2 now reported against @reinitialization. That the tag narrows @lifecycle rather than standing beside it is the trap worth naming, so it is called out in both the capability docstring and the README rather than left for an adopter to infer from a skip reason. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 28 +++++++++++++ tools/openfeature-provider-tck/spec | 2 +- .../contrib/tools/provider_tck/capability.py | 39 +++++++++++++++++++ 3 files changed, 68 insertions(+), 1 deletion(-) diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index d5f48d83c..e3d40aae8 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -161,6 +161,7 @@ SKIPPED provider does not declare capability @stale. | `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | | `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | | `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | +| `Capability.REINITIALIZATION` | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) permits rather than requires | | `Capability.TARGETING` | `@targeting` | reserved; **not declarable** — no scenarios yet | | `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | @@ -171,6 +172,33 @@ identically. Meanwhile a stateless provider has a real initialisation to verify of its own to declare `@events` for, and gating on `@events` shut it out of a scenario it should be held to. +`@reinitialization` is separate from `@lifecycle` for a subtler reason. +[Requirement 2.5.2](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) +says a provider **SHOULD** revert to its uninitialized state after `shutdown`, and its supporting +text adds that *"some providers **may** allow reinitialization from this state"*. Reuse is therefore +permitted, not required: a provider that releases its client on shutdown and declines to be started +again is exercising a choice the specification offers it, so withholding this capability needs no +`KnownDeviation` entry. + +The scenario was untagged until spec revision `fc99d5ac`, on the reading that reverting to the +uninitialized state is observable as exactly one thing — being initialisable again. That inference +does not hold, and asserting it unconditionally reported a permitted choice as a conformance failure. +A false failure is the mirror image of a vacuous pass, and this suite cares about both. Reverting the +state is not separately observable either — a provider that reverts but refuses reuse presents +identically to one that did neither — so the gated reuse scenario is the only assertion the +requirement admits. It is worth keeping for the providers that do offer reuse, because releasing the +client on shutdown while leaving an initialised flag set behind is easy to write and leaves the +provider evaluating against a closed connection rather than failing outright. + +One practical note, because it is easy to get wrong: `@reinitialization` **narrows** `@lifecycle` +rather than standing beside it. The scenario lives in `lifecycle.feature`, which carries `@lifecycle` +at the feature level, so the scenario inherits it and carries both tags — and the gate skips a +scenario when *any* capability gating it is undeclared. Reuse is therefore exercised only by an +adoption declaring `Capability.LIFECYCLE` **and** `Capability.REINITIALIZATION`; declaring the latter +alone leaves the scenario skipped on `@lifecycle` and the declaration unverified. So a provider that +withholds `LIFECYCLE` has never run this scenario, and has no evidence either way on which to declare +reuse. + Untagged scenarios are mandatory and always run. `capabilities` defaults to every *declarable* capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from the default, run the suite, and remove only what your provider genuinely cannot do. diff --git a/tools/openfeature-provider-tck/spec b/tools/openfeature-provider-tck/spec index ba002ce8e..fc99d5ace 160000 --- a/tools/openfeature-provider-tck/spec +++ b/tools/openfeature-provider-tck/spec @@ -1 +1 @@ -Subproject commit ba002ce8e807ca97920a5ebd8b9303a556f15d29 +Subproject commit fc99d5ace4da472a5fea0595fa4db8034bbbc769 diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py index b1891daf6..69beb1953 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py @@ -119,6 +119,45 @@ class Capability(str, Enum): `open-feature/spec#430 `_. """ + REINITIALIZATION = "reinitialization" + """Provider can be initialised again after ``shutdown``, and serves flags afterwards. + + Gated rather than mandatory because the specification permits reuse without + requiring it. `Requirement 2.5.2 + `_ + says a provider **SHOULD** revert to its uninitialized state after + ``shutdown``, and its supporting text adds that "some providers **may** + allow reinitialization from this state". A provider that releases its client + on shutdown and declines to be started again is exercising a choice the + specification offers it, not exhibiting a defect -- so withholding this + capability needs no :class:`~.config.KnownDeviation` entry. + + The scenario was untagged until spec revision ``fc99d5ac``, on the reading + that reverting to the uninitialized state is observable as exactly one thing + -- being initialisable again. That inference does not hold, and asserting it + unconditionally reported a permitted choice as a conformance failure. A false + failure is the mirror image of a vacuous pass. + + Reverting the state is not separately observable either: a provider that + reverts but refuses reuse presents identically to one that did neither. So + the gated reuse scenario is the only assertion the requirement admits, and it + is worth keeping for the providers that do offer reuse -- releasing the client + on shutdown while leaving an initialised flag set behind is easy to write, + and leaves the provider evaluating against a closed connection rather than + failing outright. + + **This tag narrows :attr:`LIFECYCLE` rather than standing beside it.** The + scenario lives in ``lifecycle.feature``, which carries ``@lifecycle`` at the + feature level, so the scenario inherits that tag and carries both. The gate + skips a scenario if *any* capability gating it is undeclared, so reuse is + exercised only by an adoption declaring :attr:`LIFECYCLE` **and** this -- + declaring this one alone leaves the scenario skipped on ``@lifecycle``, and + the declaration unverified. Which is the trap worth naming: a provider that + withholds ``LIFECYCLE`` never ran this scenario, at this pin or the one + before it, so nothing about its behaviour on reuse has been observed either + way and there is no evidence on which to declare this. + """ + TARGETING = "targeting" """Reserved, and **not declarable**. No scenario carries this tag: targeting is backend evaluation logic.""" From 88b754a3ed0411217146f9082fe2517650a98479 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 16:58:41 +0200 Subject: [PATCH 15/46] fix(provider-tck): require the SDK release that can await a provider The suite calls `api.set_provider_and_wait` so that a scenario evaluates a flag only once the provider has initialised. That function arrived in openfeature-sdk 0.10.0, but the package still declared `>=0.8.2`, so the workspace lock resolved 0.8.4 and every scenario died on AttributeError: module 'openfeature.api' has no attribute 'set_provider_and_wait' CI runs `uv sync --frozen`, so it installed the locked 0.8.4 and saw the same failure rather than the green suite the branch claims. Raise the floor to the release that actually carries the function and relock. Only openfeature-sdk moves, 0.8.4 -> 0.10.0. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/pyproject.toml | 2 +- uv.lock | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/tools/openfeature-provider-tck/pyproject.toml b/tools/openfeature-provider-tck/pyproject.toml index ff0cbe43a..84ff72786 100644 --- a/tools/openfeature-provider-tck/pyproject.toml +++ b/tools/openfeature-provider-tck/pyproject.toml @@ -17,7 +17,7 @@ classifiers = [ ] keywords = ["openfeature", "conformance", "tck", "feature-flags"] dependencies = [ - "openfeature-sdk>=0.8.2", + "openfeature-sdk>=0.10.0", "pytest>=8.4.0", # Same runner the flagd provider and the flagd testkit already use, so an # adopting module gains no new test framework. diff --git a/uv.lock b/uv.lock index b0ab379c1..804640a93 100644 --- a/uv.lock +++ b/uv.lock @@ -844,7 +844,7 @@ name = "exceptiongroup" version = "1.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/50/79/66800aadf48771f6b62f7eb014e352e5d06856655206165d775e675a02c9/exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219", size = 30371, upload-time = "2025-11-21T23:01:54.787Z" } wheels = [ @@ -2009,7 +2009,7 @@ dev = [ [package.metadata] requires-dist = [ - { name = "openfeature-sdk", specifier = ">=0.8.2" }, + { name = "openfeature-sdk", specifier = ">=0.10.0" }, { name = "pytest", specifier = ">=8.4.0" }, { name = "pytest-bdd", specifier = ">=8.1.0,<9.0.0" }, ] @@ -2109,11 +2109,11 @@ dev = [ [[package]] name = "openfeature-sdk" -version = "0.8.4" +version = "0.10.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/3b/08/f6698d0614b8703170117b786bd77b7b0a04f3ee00f19fbe9b360d2dee69/openfeature_sdk-0.8.4.tar.gz", hash = "sha256:66abf71f928ec8c0db1111072bb0ef2635dfbd09510f77f4b548e5d0ea0e6c1a", size = 29676, upload-time = "2025-12-09T07:31:13.137Z" } +sdist = { url = "https://files.pythonhosted.org/packages/24/30/cfc684b7d8314398d476ae8ed515c10db99c4d7f950989db464b4ded12ce/openfeature_sdk-0.10.0.tar.gz", hash = "sha256:938c2540bdea4da3b01ef507517ee636f223a35abaaca845c5587e594151b052", size = 33516, upload-time = "2026-06-01T19:45:35.136Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/9c/80/f6532778188c573cc83790b11abccde717d4c1442514e722d6bb6140e55c/openfeature_sdk-0.8.4-py3-none-any.whl", hash = "sha256:805ba090669798fc343ca9fdcbc56ff0f4b57bf6757533f0854d2021192e620a", size = 35986, upload-time = "2025-12-09T07:31:12.092Z" }, + { url = "https://files.pythonhosted.org/packages/da/44/8a4f5225e930ff0d999fd43f5d743a4babeb6c7e76dddc00f0e118878ef3/openfeature_sdk-0.10.0-py3-none-any.whl", hash = "sha256:75497ea75d73f684eef509a25f79ad6386368862e050af80ab70a44ae49b33e4", size = 38941, upload-time = "2026-06-01T19:45:33.011Z" }, ] [[package]] From 60054b09bda74507e101c19e114f05e806cd3218 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 18:05:44 +0200 Subject: [PATCH 16/46] fix(provider-tck): identify a canonical feature the way the other languages do An audit across the four language suites found three different canonical uri forms for the same file: gherkin/errors.feature Go specification/assets/provider-tck/gherkin/errors.feature JavaScript features/errors.feature this suite A consumer joining two languages' results keys on the uri and the scenario name, so the partition Appendix F's rule exists to guarantee is precisely the thing that did not survive it. Appendix F now states the form rather than implying it: a canonical feature is identified by its path *relative to the asset directory* -- gherkin/errors.feature -- and an extension mounts under extensions/. The structural cause is a single local rename. Go consumes the assets as a Go module whose root *is* the asset directory, so its embed keys are gherkin/*.feature and it gets the right uri for nothing. The other three vendor the assets into a locally-named directory, and the uri inherits that local name. Here the name was CANONICAL_DIRECTORY = "features", which the sync copied gherkin/ into and which uri_for() then reported. Point it at the name the assets already have and the copy, the reserved prefix and the emitted uri agree again. The reported uri looked right in one place and was wrong in another, which is worth recording: uri_for() takes precedence over pytest-bdd's rel_filename at the emitter's call site, so the pytest-bdd path is only a fallback. Reading the fallback alone suggests this suite was already correct. EXTENSIONS_DIRECTORY moves from "tck-extensions" to "extensions" in the same commit, because Java's TCK is being renamed the same way in the same round and the docstring's parity claim -- that an adopter shipping a provider in both languages puts the same directory in both repositories -- is only true if both move. The other half of that argument still holds and is now stated rather than assumed: an extensions directory must not share the canonical name, because a directory sharing it is how an extension comes to occupy a canonical file's identity, and "gherkin" and "extensions" are distinct. EXTENSIONS_URI_PREFIX stays a constant of its own even though it now equals EXTENSIONS_DIRECTORY. The directory this suite scans and the prefix a report is keyed by are two facts, and only the second is fixed by Appendix F. One name doing both jobs is exactly what went wrong on the canonical half. reserved_prefix_problem() and collision_problem() build their messages from the constants, so they follow the rename rather than policing a stale string; the same is true of is_canonical_uri() and uri_collisions(). The gitignore entry and the packaged-wheel artifact list name the copied directory literally and move with it. One test is added. Every existing assertion is written against the constants, so it holds whatever they say -- renaming one would leave the suite green while the uris stopped joining with another language's, which is the failure that happened. The two strings Appendix F fixes are now pinned as literals. The spec assets themselves did not change, so the submodule pin does not move. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/.gitignore | 2 +- tools/openfeature-provider-tck/README.md | 20 ++++--- .../hatch_build_sync.py | 9 ++- tools/openfeature-provider-tck/pyproject.toml | 2 +- .../contrib/tools/provider_tck/__init__.py | 4 +- .../contrib/tools/provider_tck/extensions.py | 57 ++++++++++++------- .../tests/test_extensions.py | 39 ++++++++++--- 7 files changed, 91 insertions(+), 42 deletions(-) diff --git a/tools/openfeature-provider-tck/.gitignore b/tools/openfeature-provider-tck/.gitignore index 066646223..72a17508a 100644 --- a/tools/openfeature-provider-tck/.gitignore +++ b/tools/openfeature-provider-tck/.gitignore @@ -2,6 +2,6 @@ # DO NOT EDIT the copies, and do not commit them: the canonical definitions live # in spec/specification/assets/provider-tck/, and the revision this package is # built against is recorded by the submodule pin. -src/openfeature/contrib/tools/provider_tck/features/ +src/openfeature/contrib/tools/provider_tck/gherkin/ src/openfeature/contrib/tools/provider_tck/flag_data/ src/openfeature/contrib/tools/provider_tck/control-api.yaml diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index e3d40aae8..39765c3b6 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -77,14 +77,14 @@ proprietary rollout rule, and the behaviour of those is as worth pinning as the top of. Verifying them used to mean a second harness: a second backend lifecycle, a second set of fixtures, a second thing to keep working. -Put them in the same run instead. Create a directory named `tck-extensions` beside the module that +Put them in the same run instead. Create a directory named `extensions` beside the module that calls `scenarios()`, and write step definitions for whatever is new in a `conftest.py` beside it: ``` tests/ ├── conftest.py # your step definitions ├── test_conformance.py # the fixture and the one call, unchanged -└── tck-extensions/ +└── extensions/ └── fractional.feature ``` @@ -114,18 +114,20 @@ scenarios(*feature_paths()) That line does not change when you add an extension, and it is the only difference from `scenarios(features_path())` — which still works and still sees only the canonical set. An adopter -with no `tck-extensions` directory runs exactly what they ran before: same scenarios, same count. +with no `extensions` directory runs exactly what they ran before: same scenarios, same count. ### Your scenarios cannot stand in for ours Every feature file carries a uri, and it is how a canonical scenario is told from an adopter's: -canonical files are the ones under the `features/` prefix and yours are under `extensions/` — the +canonical files are the ones under the `gherkin/` prefix and yours are under `extensions/` — the prefix Go and JavaScript mount theirs under too, so a consumer holding conformance reports from -several languages applies one rule. The prefix is derived from where a file *is*, not from what the -runner called it, and `extensions.py` reports two cases that derivation cannot rule out: +several languages applies one rule. Neither prefix is this package's to choose: Appendix F +identifies a canonical feature by its path *relative to the specification's asset directory*, which +is what makes it `gherkin/`. The prefix is derived from where a file *is*, not from what the runner +called it, and `extensions.py` reports two cases that derivation cannot rule out: -- **A feature file of yours under the reserved `features/` prefix.** Handing `scenarios()` a - directory of your own named `features` is the one route left to a canonical-looking uri. +- **A feature file of yours under the reserved `gherkin/` prefix.** Handing `scenarios()` a + directory of your own named `gherkin` is the one route left to a canonical-looking uri. - **Two feature files that would share one uri.** A record of what ran holds one copy of a feature file per uri, so the second file's scenarios would be attributed to the first file's. @@ -133,7 +135,7 @@ This is not hypothetical. Java's suite found that a same-named feature file in a root *replaced* the canonical one, and the run went green having asked the adopter's questions instead of the specification's — the worst outcome available to a conformance suite. The Python route to the same place is narrower and just as quiet: pytest-bdd names a feature file by its parent -directory joined to its own name, so `tck-extensions/features/errors.feature` arrives under the uri +directory joined to its own name, so `extensions/gherkin/errors.feature` arrives under the uri the canonical `errors.feature` already occupies. ## Capabilities diff --git a/tools/openfeature-provider-tck/hatch_build_sync.py b/tools/openfeature-provider-tck/hatch_build_sync.py index f31bc55b5..102590882 100644 --- a/tools/openfeature-provider-tck/hatch_build_sync.py +++ b/tools/openfeature-provider-tck/hatch_build_sync.py @@ -25,7 +25,14 @@ ) # (source directory or file, destination) relative to SPEC_ASSETS / DEST_BASE. -TREES = [("gherkin", "features"), ("flags", "flag_data")] +# +# "gherkin" copies to a directory of the same name on purpose, and the pair is not +# redundant: a canonical feature is identified by its path relative to the asset +# directory, so the destination name *is* the reported uri prefix. Renaming it +# locally -- it used to land in "features" -- silently renamed the uri, which is +# how this suite reported features/errors.feature for the file Go reports as +# gherkin/errors.feature. Keep the two equal. +TREES = [("gherkin", "gherkin"), ("flags", "flag_data")] FILES = [("openapi/control-api.yaml", "control-api.yaml")] diff --git a/tools/openfeature-provider-tck/pyproject.toml b/tools/openfeature-provider-tck/pyproject.toml index 84ff72786..3825d97b5 100644 --- a/tools/openfeature-provider-tck/pyproject.toml +++ b/tools/openfeature-provider-tck/pyproject.toml @@ -55,7 +55,7 @@ packages = ["src/openfeature"] # Ship the conformance assets even though they are gitignored: an adopter # installing this package must need no submodule of their own. artifacts = [ - "src/openfeature/contrib/tools/provider_tck/features/", + "src/openfeature/contrib/tools/provider_tck/gherkin/", "src/openfeature/contrib/tools/provider_tck/flag_data/", "src/openfeature/contrib/tools/provider_tck/control-api.yaml", ] diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py index ed361a34b..37635a1e7 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py @@ -35,7 +35,7 @@ def tck_config(): injects the generated tests into the *calling module* by walking the stack, so a convenience wrapper around it would deposit them inside this package instead. :func:`~.extensions.feature_paths` is the canonical assets plus a -``tck-extensions`` directory beside the calling module, if there is one -- see +``extensions`` directory beside the calling module, if there is one -- see :mod:`~.extensions`. The step definitions arrive through this package's pytest plugin, so there is @@ -95,7 +95,7 @@ def tck_config(): # NOTE ON THE SOURCE OF TRUTH # -# The files under features/ and flag_data/, and control-api.yaml, are NOT owned +# The files under gherkin/ and flag_data/, and control-api.yaml, are NOT owned # by this repository and are NOT committed to it. They are copies of the # language-agnostic conformance artifacts defined in open-feature/spec under # specification/assets/provider-tck/, which reaches this package as a git diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py index e5796d64e..26849b811 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py @@ -15,7 +15,7 @@ adopter's ``conftest.py`` -- or in the test module itself -- is in scope for the scenarios ``scenarios()`` generates there. The only thing pytest cannot find by itself is the feature files, which is what this module finds: a directory named -``tck-extensions`` beside the adopter's test module. +``extensions`` beside the adopter's test module. That leaves one line, and it is the same line whether or not there are extensions:: @@ -26,13 +26,13 @@ apart by the uri each feature file is identified by, and this module derives that uri from where the file *is* rather than taking what the runner offers: -* ``features/…`` is the packaged canonical assets, and nothing else; +* ``gherkin/…`` is the packaged canonical assets, and nothing else; * ``extensions/…`` is a discovered extension, whatever the adopter's own - directory layout under ``tck-extensions`` looks like. + directory layout under ``extensions`` looks like. The derivation is not decoration. pytest-bdd names a feature file by its parent -directory joined to its own name, so ``tck-extensions/features/errors.feature`` -arrives as ``features/errors.feature`` -- the same uri as a canonical file. A +directory joined to its own name, so ``extensions/gherkin/errors.feature`` +arrives as ``gherkin/errors.feature`` -- the same uri as a canonical file. A record of what ran holds one copy of a feature file per uri, so the second file is never read and its scenarios are attributed to the first one's or to nothing at all. Java hit the same thing by a different route: a same-named feature file @@ -70,23 +70,34 @@ _PACKAGE = "openfeature.contrib.tools.provider_tck" -CANONICAL_DIRECTORY = "features" +CANONICAL_DIRECTORY = "gherkin" """The packaged directory the canonical feature files live in. Also the uri prefix they are identified by, which is why it is reserved: anyone -reading ``features/errors.feature`` is entitled to assume it is the +reading ``gherkin/errors.feature`` is entitled to assume it is the specification's file rather than a local one that happened to land in a directory of that name. + +The name is no longer chosen here. Appendix F fixes it: a canonical feature is +identified by its path **relative to the specification's asset directory**, and +``gherkin`` is the directory it occupies there. This suite used to vendor those +assets under a local name of its own and report that name instead, which is how +it came to answer ``features/errors.feature`` where Go -- consuming the same +assets as a module whose root *is* that directory -- answered +``gherkin/errors.feature``. A consumer joining two languages' results keys on the +uri and the scenario name, so the local name was the whole of the divergence. """ -EXTENSIONS_DIRECTORY = "tck-extensions" +EXTENSIONS_DIRECTORY = "extensions" """Where an adopter puts feature files of their own, beside their test module. -Deliberately not ``features``: a directory sharing the canonical name is how an -extension comes to occupy a canonical file's identity, and a convention that -cannot collide is worth more than one that reads slightly better. The name is -the one Java's TCK scans for on the classpath, so an adopter who ships a provider -in both languages puts the same directory in both repositories. +Deliberately not the canonical name: a directory sharing it is how an extension +comes to occupy a canonical file's identity, and a convention that cannot collide +is worth more than one that reads slightly better. ``gherkin`` and ``extensions`` +are distinct, so that still holds. The name is the one Java's TCK scans for on the +classpath -- renamed to ``extensions`` there in the same round as here -- so an +adopter who ships a provider in both languages still puts the same directory in +both repositories. """ EXTENSIONS_URI_PREFIX = "extensions" @@ -95,6 +106,12 @@ The Go and JavaScript suites mount extensions under the same prefix, so a consumer holding reports from several languages applies one rule to tell an adopter's scenario from the specification's. + +Equal to :data:`EXTENSIONS_DIRECTORY` today, and still a constant of its own: the +directory this suite scans and the prefix a report is keyed by are two separate +facts, and only the second is fixed by Appendix F. Collapsing them is exactly what +went wrong on the canonical half, where one name did both jobs and the reported +uri inherited a local choice. """ @@ -111,7 +128,7 @@ def features_path() -> str: def feature_paths() -> tuple[str, ...]: """Return every feature directory this adoption should run. - The canonical set, always, and a ``tck-extensions`` directory beside the + The canonical set, always, and an ``extensions`` directory beside the calling module if there is one. Hand the result to pytest-bdd's ``scenarios()``:: @@ -185,8 +202,8 @@ def uri_for(path: Path) -> str | None: Derived from the file's location rather than from pytest-bdd's ``rel_filename``, which is the parent directory's name joined to the file's - own. That is what let ``tck-extensions/features/errors.feature`` present - itself as ``features/errors.feature``: the same uri as a canonical file, and + own. That is what let ``extensions/gherkin/errors.feature`` present + itself as ``gherkin/errors.feature``: the same uri as a canonical file, and a record of what ran holds one copy of a feature file per uri. """ resolved = _resolve(path) @@ -205,7 +222,7 @@ def reserved_prefix_problem(uri: str, path: Path) -> str | None: """Report a feature file claiming the canonical uri prefix without being canonical. The one thing the naming convention cannot rule out on its own: an adopter - who hands ``scenarios()`` a directory of their own named ``features``. The + who hands ``scenarios()`` a directory of their own named ``gherkin``. The file is then named exactly as a canonical one would be, and a reader has no way to tell that the specification did not write it. @@ -232,9 +249,9 @@ def uri_collisions( Deriving the uri from the file's location removes the collision an adopter is actually likely to hit, but it does not make one impossible. Two extension roots contributing the same relative path to a single suite -- two - test modules sharing one ``tck_config`` from a conftest, each with a - ``tck-extensions/vendor.feature`` -- still land on ``extensions/vendor.feature`` - twice, and so does a ``tck-extensions`` directory nested inside another one. + test modules sharing one ``tck_config`` from a conftest, each with an + ``extensions/vendor.feature`` -- still land on ``extensions/vendor.feature`` + twice, and so does an ``extensions`` directory nested inside another one. That has to be refused rather than resolved. A record of what ran holds one copy of a feature file per uri, so the second file is never read: its diff --git a/tools/openfeature-provider-tck/tests/test_extensions.py b/tools/openfeature-provider-tck/tests/test_extensions.py index 8d0b407e6..a6e95d65e 100644 --- a/tools/openfeature-provider-tck/tests/test_extensions.py +++ b/tools/openfeature-provider-tck/tests/test_extensions.py @@ -19,7 +19,7 @@ went green having asked the adopter's questions instead of the specification's. The Python route to the same place is narrower and just as quiet: pytest-bdd names a feature file by its parent directory joined to its own name, so a file at -``tck-extensions/features/errors.feature`` arrives under the uri the canonical +``extensions/gherkin/errors.feature`` arrives under the uri the canonical ``errors.feature`` already occupies. The first three are properties of how a whole session runs rather than of what a @@ -341,7 +341,7 @@ def test_features_path_sees_no_extension_however_many_are_beside_it( ) -> None: """The older call still means exactly what it meant: the canonical set. - Both generated suites sit in the same directory as the ``tck-extensions`` + Both generated suites sit in the same directory as the ``extensions`` directory, so the one that asks for ``features_path()`` is asking with an extension in arm's reach and must still not see it. """ @@ -355,7 +355,7 @@ def test_features_path_sees_no_extension_however_many_are_beside_it( def test_feature_paths_is_the_canonical_set_when_there_is_no_extension_directory() -> ( None ): - """This test module has no ``tck-extensions`` beside it, and gets one path.""" + """This test module has no ``extensions`` beside it, and gets one path.""" assert not (Path(__file__).parent / EXTENSIONS_DIRECTORY).exists() assert feature_paths() == (features_path(),) @@ -390,11 +390,32 @@ def test_the_canonical_features_are_found_inside_the_distribution() -> None: # -- deriving the uri -------------------------------------------------------- +def test_the_two_prefixes_are_the_ones_appendix_f_names() -> None: + """Pinned as literals, because every other assertion here uses the constants. + + Those assertions hold whatever the constants say, so renaming one would leave + the suite green while the uris it emits stopped joining with another + language's -- which is the failure that happened. Appendix F fixes both + strings: a canonical feature is identified by its path relative to the + specification's asset directory, and ``gherkin`` is the directory it occupies + there; an extension mounts under ``extensions``. + """ + assert CANONICAL_DIRECTORY == "gherkin" + assert EXTENSIONS_URI_PREFIX == "extensions" + assert CANONICAL_DIRECTORY != EXTENSIONS_DIRECTORY, ( + "an extensions directory sharing the canonical name is how an extension " + "comes to occupy a canonical file's identity" + ) + + def test_the_canonical_assets_keep_the_reserved_prefix() -> None: canonical = Path(features_path()) / CANONICAL_FEATURE assert uri_for(canonical) == f"{CANONICAL_DIRECTORY}/{CANONICAL_FEATURE}" assert is_canonical_uri(f"{CANONICAL_DIRECTORY}/{CANONICAL_FEATURE}") - assert reserved_prefix_problem(f"features/{CANONICAL_FEATURE}", canonical) is None + assert ( + reserved_prefix_problem(f"{CANONICAL_DIRECTORY}/{CANONICAL_FEATURE}", canonical) + is None + ) def test_an_extension_keeps_its_layout_below_the_extensions_prefix( @@ -404,7 +425,7 @@ def test_an_extension_keeps_its_layout_below_the_extensions_prefix( Including one that reproduces the canonical name, which is the collision the derivation exists for: pytest-bdd would have called the second of these - ``features/errors.feature``. + ``gherkin/errors.feature``. """ root = tmp_path / EXTENSIONS_DIRECTORY assert uri_for(root / "vendor.feature") == "extensions/vendor.feature" @@ -415,7 +436,9 @@ def test_an_extension_keeps_its_layout_below_the_extensions_prefix( assert ( uri_for(root / "a" / "b" / "vendor.feature") == "extensions/a/b/vendor.feature" ) - assert not is_canonical_uri(f"extensions/features/{CANONICAL_FEATURE}") + assert not is_canonical_uri( + f"{EXTENSIONS_URI_PREFIX}/{CANONICAL_DIRECTORY}/{CANONICAL_FEATURE}" + ) def test_a_derived_uri_is_slash_separated_on_every_platform(tmp_path: Path) -> None: @@ -440,7 +463,7 @@ def test_a_local_file_under_the_reserved_prefix_is_a_problem(tmp_path: Path) -> """The one route to a canonical-looking uri the convention cannot close. An adopter may still hand ``scenarios()`` a directory of their own named - ``features``, and its files are then named exactly as canonical ones would + ``gherkin``, and its files are then named exactly as canonical ones would be. Reported rather than raised: the scenarios are the adopter's to run, and it is publishing them as the specification's that has to be refused. """ @@ -454,7 +477,7 @@ def test_a_local_file_under_the_reserved_prefix_is_a_problem(tmp_path: Path) -> def test_two_files_that_would_share_one_uri_are_reported(tmp_path: Path) -> None: """Deriving the uri from the location narrows the collision; it does not end it. - A ``tck-extensions`` directory nested inside another one reaches the same uri + An ``extensions`` directory nested inside another one reaches the same uri as its namesake at the root, and so would two test modules sharing one ``tck_config``. """ From d68e8ec8a665b57fc7b10f5efac04e5ab013f31b Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 18:35:33 +0200 Subject: [PATCH 17/46] feat(provider-tck): ship the HTTP control client with the suite The control API is normative -- Appendix F defines it as an HTTP surface a backend under test MUST expose -- so every adoption that drives a real backend needs a client for it. With the client in the flagd adoption the suite shipped the contract and not the thing that speaks it, and an adopter taking this package had to write their own. It went unnoticed because the only other adoption, OFREP, is stacked on flagd and inherited it. A third-party adopter is the case nobody was standing in for. Nothing about it was flagd-specific: urllib.request only, so the suite still gains no HTTP client dependency and no container dependency, and DEFAULT_CONFIGURATION is the configuration name Appendix F requires of every backend. Its documentation and its eighteen tests come with it; orchestrating a stack and discovering its mapped ports stays with the adopter, which is the part that is genuinely vendor-specific. Java already shipped its equivalent on the suite side; Go moved in the same round. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 41 ++- .../contrib/tools/provider_tck/__init__.py | 8 + .../contrib/tools/provider_tck/httpcontrol.py | 274 ++++++++++++++++++ .../tests/test_http_control.py | 234 +++++++++++++++ 4 files changed, 548 insertions(+), 9 deletions(-) create mode 100644 tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/httpcontrol.py create mode 100644 tools/openfeature-provider-tck/tests/test_http_control.py diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index 39765c3b6..1af1bea08 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -280,11 +280,31 @@ definitions never talk to a backend directly, which is why the same Gherkin runs containerised backend and against a provider manipulated in-process. **If your provider talks to a backend, drive it over the HTTP control API** — the document is -available as `control_api_spec()`. That API is the normative contract for those providers, and it is -what makes a conformance claim portable: another language's TCK drives the same endpoints against -the same stack and must get the same answers. +available as `control_api_spec()`, and `HttpControl` is the client for it. That API is the normative +contract for those providers, and it is what makes a conformance claim portable: another language's +TCK drives the same endpoints against the same stack and must get the same answers. -Two of its requirements are easy to get wrong: +```python +control = HttpControl(f"http://localhost:{container.get_launchpad_port()}") +``` + +`HttpControl` is built on `urllib.request` alone, so the TCK gains no HTTP client and no container +dependency. **Orchestrating the stack stays with you**, where the vendor-specific knowledge already +lives — which compose file, which services, which internal ports. That is a deliberate trade against +the "provider authors write no test infrastructure" goal, and worth revisiting once a second +containerised adopter shows what is actually common. + +Two of its behaviours are worth knowing about: + +- **`/reset` is optional and the fallback is automatic.** `prepare_scenario()` prefers `POST /reset`, + which restores the flag baseline with no availability blip; a backend without it answers 404 or + 501 and the client falls back to `POST /start?config=default`. The probe happens once per suite. + flagd-testbed's launchpad registers only `/start`, `/restart`, `/stop` and `/change`, so that + fallback is the normal path today. +- **After a disconnect it starts rather than resets.** `/reset` restores flag *state*; it is not + specified to bring a stopped backend back up. + +Two of the API's requirements are easy to get wrong: - **Containers are never stopped or restarted mid-suite.** Unavailability is simulated *inside* the running stack. Container orchestrators assign host ports dynamically and cannot reliably preserve @@ -385,14 +405,15 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_lifecycle_steps` | the steps that call the provider directly | the in-memory suites skip `@lifecycle`, so the shutdown, re-initialise and metadata steps are driven against a recording provider instead | | `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | | `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity | +| `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping and the control-API it reports, against a stubbed control API | ``` -106 passed, 21 skipped, 2 xfailed +125 passed, 21 skipped, 2 xfailed ``` -No Docker and no network. The conformance suites take under a second; `test_extensions` takes most -of the rest, because the properties it checks are properties of a whole pytest session and it runs a -generated adoption in a subprocess to check them. +No Docker and no network beyond loopback. The conformance suites take under a second; +`test_extensions` takes most of the rest, because the properties it checks are properties of a whole +pytest session and it runs a generated adoption in a subprocess to check them. Neither in-memory suite declares `@lifecycle`, so the six lifecycle scenarios — three about initialisation, three about shutdown — are skipped in both. That is the point: with no backend to @@ -404,7 +425,9 @@ finding 3, so its three scenarios are skipped too. - **Evaluation context passthrough is unverifiable.** The scenarios build evaluation contexts but cannot assert one *reached* the backend. That needs an echo operation on the control API. -- **No HTTP control client yet.** It arrives with the first containerised adopter. +- **No shared containerised-backend helper.** `HttpControl` drives the control API, but starting the + stack and discovering its mapped ports is still each adopter's own code. Abstracting that from a + single example tends to produce the wrong abstraction; it should wait for a second adopter. - **Caching, hooks and flag metadata** are not covered. [appendix-a]: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py index 37635a1e7..20e415d88 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py @@ -64,6 +64,11 @@ def tck_config(): feature_paths, features_path, ) +from .httpcontrol import ( + DEFAULT_CONFIGURATION, + ControlApiError, + HttpControl, +) from .inprocess import InProcessControl from .provider import ( CHANGING_FLAG_KEY, @@ -75,12 +80,15 @@ def tck_config(): __all__ = [ "CHANGING_FLAG_KEY", "DECLARABLE_CAPABILITIES", + "DEFAULT_CONFIGURATION", "EXTENSIONS_DIRECTORY", "RESERVED_CAPABILITIES", "BackendControl", "Capability", "ConnectionControl", + "ControlApiError", "ControllableInMemoryProvider", + "HttpControl", "InProcessControl", "KnownDeviation", "TckConfig", diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/httpcontrol.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/httpcontrol.py new file mode 100644 index 000000000..5e787b5e7 --- /dev/null +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/httpcontrol.py @@ -0,0 +1,274 @@ +"""HTTP backend control: the normative control path for a provider with a real backend.""" + +from __future__ import annotations + +import threading +import typing +import urllib.error +import urllib.parse +import urllib.request + +from .control import BackendControl, ConnectionControl + +__all__ = ["DEFAULT_CONFIGURATION", "ControlApiError", "HttpControl"] + +DEFAULT_CONFIGURATION = "default" +"""The configuration name every backend under test must support. + +It is the one that serves the canonical flag set the feature files assume. +""" + +DEFAULT_TIMEOUT = 30.0 +"""Seconds bounding a single control-API request. + +Control calls are local HTTP to a container on the same host; anything slower +than this is a wedged backend rather than a slow one. +""" + +_NOT_IMPLEMENTED = frozenset({404, 501}) +"""How a backend that does not implement ``/reset`` answers it, per the OpenAPI document.""" + +_SUPPORTED_SCHEMES = frozenset({"http", "https"}) + + +class ControlApiError(RuntimeError): + """Raised when a control-API call fails or answers with an unexpected status. + + Always a defect in the stack under test or in its wiring, never a provider + defect -- so it is raised rather than swallowed. A control call that quietly + did nothing would leave the next scenario running against an unknown backend + state and reporting whatever it found as a conformance result. + """ + + +class HttpControl: + """Drives a backend under test over the HTTP control API in ``control-api.yaml``. + + This is the normative control path for any provider with a real backend, and + it is what makes a conformance claim portable: another language's TCK drives + the same endpoints against the same stack and must get the same answers. + + Built on :mod:`urllib.request` alone, so adopting the TCK pulls in no HTTP + client and no container library. Orchestrating the stack stays with the + adopting suite, where the vendor-specific knowledge already lives -- which + compose file, which services, which internal ports. + + **What it never does.** It never stops, kills or recreates a container. + Unavailability is simulated inside the running stack, through ``POST /stop``, + because container orchestrators assign host ports dynamically and cannot + reliably preserve them across a restart: a restarted backend generally comes + back on a different host port, silently invalidating every provider already + pointed at the old one, and the resulting failure looks like a flaky provider + rather than a broken test. Starting and stopping the stack itself belongs to + the adopting suite, once per session. + + **Scenario isolation.** :meth:`prepare_scenario` prefers ``POST /reset``, + which restores the flag baseline with no availability blip and therefore + cannot inject a spurious lifecycle event into the next scenario. That + operation is optional, and a backend that does not implement it answers 404 + or 501; the TCK then falls back to ``POST /start?config=...``, which also + resets flag state at the cost of a process restart. The fallback is probed + once and remembered for the rest of the suite. + + **After a disconnect, ``/start`` rather than ``/reset``.** ``/reset`` is + specified to restore flag state, not to bring a stopped backend back up, so + a disconnect is recorded and the scenario that follows one is prepared with + ``/start``. + + Safe to share between suites, and it should be shared whenever they drive the + same backend: the disconnect bookkeeping is only correct if every operation + against one backend goes through one instance of this class. + """ + + def __init__( + self, + base_url: str, + *, + configuration: str = DEFAULT_CONFIGURATION, + timeout: float = DEFAULT_TIMEOUT, + ) -> None: + """Build a control for the backend whose control API is rooted at ``base_url``. + + :param base_url: root of the control API, for example + ``http://localhost:32768``. It must be built from the dynamically + mapped host port of the control service, discovered after the stack + is up -- a stack under test must not pin host ports. + :param configuration: the named flag configuration to seed. Defaults to + :data:`DEFAULT_CONFIGURATION`, the only name every backend must + support and the one serving the canonical flag set. + :param timeout: seconds bounding a single control-API request. + """ + parsed = urllib.parse.urlsplit(base_url) + if parsed.scheme not in _SUPPORTED_SCHEMES or not parsed.netloc: + msg = ( + f"base_url {base_url!r} is not an http(s) URL. It is the root of the " + f"control API, built from the dynamically mapped host port of the " + f"control service, for example 'http://localhost:32768'" + ) + raise ValueError(msg) + + self._base_url = base_url.rstrip("/") + self._configuration = configuration + self._timeout = timeout + + self._lock = threading.Lock() + # None until the first /reset call tells us which way it went. + self._reset_supported: bool | None = None + # Set by any operation that may have left the backend down, so the next + # prepare_scenario starts it rather than merely resetting flag state. + self._backend_maybe_down = False + + @property + def control_api(self) -> str: + """Report that this control drives its backend over the HTTP control API. + + The optional property ``BackendControl`` documents. This is the one + control in the package that can answer it without qualification: every + operation below is an HTTP request to ``control-api.yaml``. Leaving it + unsaid would put a report from the normative control path on the same + footing as one from a control that declined to say which path it took. + """ + return "http" + + @property + def description(self) -> str: + return f"the backend at {self._base_url}, driven over the control API" + + def prepare_scenario(self) -> None: + """Bring the backend to the state every scenario starts from. + + Prefers ``/reset`` and falls back to ``/start`` -- see the class + documentation for why, and for why a disconnect forces ``/start``. + """ + with self._lock: + must_start = self._backend_maybe_down or self._reset_supported is False + + if must_start: + self._start() + return + + status = self._call("/reset") + + if status in _NOT_IMPLEMENTED: + # The documented fallback. Remembered so the probe costs one request + # per suite rather than one per scenario. + with self._lock: + self._reset_supported = False + self._start() + return + + if not self._is_success(status): + msg = f"POST /reset on {self._base_url} returned {status}" + raise ControlApiError(msg) + + with self._lock: + self._reset_supported = True + + def change_flag(self) -> None: + """Mutate flag configuration so a conforming provider observes a change.""" + self._require("/change") + + def disconnect(self) -> None: + """Make the backend unreachable, without touching any container. + + The backend *process* inside the still-running container is stopped. See + the class documentation for why that distinction is a requirement rather + than a preference. + """ + with self._lock: + self._backend_maybe_down = True + self._require("/stop") + + def reconnect(self) -> None: + """Make the backend reachable again, preserving flag state. + + Starting with the configuration already in effect restores the same + baseline, so the provider observes a change in availability and never a + change in flag values. + """ + self._start() + + def restart(self, seconds: int) -> None: + """Take the backend down for ``seconds`` and bring it back. + + Part of the control API rather than of :class:`~.control.ConnectionControl`: + no scenario drives a bounded outage today, because + :meth:`disconnect`/:meth:`reconnect` let a scenario end the outage when + it is ready instead of guessing how long a provider needs to notice one. + Exposed because the operation is required of every backend and an + adopting suite may want it for its own tests. + + Unlike ``/stop`` followed by ``/start``, this preserves flag state + across the outage. + """ + with self._lock: + self._backend_maybe_down = True + self._require("/restart", {"seconds": str(seconds)}) + with self._lock: + self._backend_maybe_down = False + + def _start(self) -> None: + self._require("/start", {"config": self._configuration}) + with self._lock: + self._backend_maybe_down = False + + def _require(self, path: str, query: dict[str, str] | None = None) -> None: + """Perform a control call and fail on any non-2xx response.""" + status = self._call(path, query) + if not self._is_success(status): + msg = f"POST {path} on {self._base_url} returned {status}" + raise ControlApiError(msg) + + def _call(self, path: str, query: dict[str, str] | None = None) -> int: + """Perform one control-API request and return its status code. + + The response body is read and discarded: the control API's bodies are + human-readable messages the TCK is specified never to interpret, and + reading them lets the connection be released cleanly. + """ + target = self._base_url + path + if query: + target += "?" + urllib.parse.urlencode(query) + + # An empty body rather than none, so the request carries Content-Length + # even where a proxy in the stack insists on one. + # + # S310 wants the scheme audited before a URL is opened; __init__ rejects + # any base_url that is not http(s), and target is built from that + # validated base URL plus a literal path, so no other scheme can reach + # here. + request = urllib.request.Request(target, data=b"", method="POST") # noqa: S310 + + try: + with urllib.request.urlopen(request, timeout=self._timeout) as response: # noqa: S310 + response.read() + return int(response.status) + except urllib.error.HTTPError as error: + # A status the server chose to report as an error is still an answer, + # and /reset answering 404 is the documented way to say "not + # implemented" -- so this is a return, not a raise. + with error: + error.read() + return int(error.code) + except OSError as error: + msg = ( + f"control request POST {target} failed: {error}. The control API must " + f"stay reachable even while the backend is deliberately down, " + f"otherwise an outage cannot be ended" + ) + raise ControlApiError(msg) from error + + @staticmethod + def _is_success(status: int) -> bool: + return 200 <= status < 300 + + +if typing.TYPE_CHECKING: + # Static assertion, erased at runtime: HttpControl must satisfy both control + # protocols, the way Go's `var _ BackendControl = (*HTTPControl)(nil)` does. + # A method renamed out of the protocol fails type-checking rather than at the + # first scenario that needs it. + def _implements( + control: HttpControl, + ) -> tuple[BackendControl, ConnectionControl]: + return control, control diff --git a/tools/openfeature-provider-tck/tests/test_http_control.py b/tools/openfeature-provider-tck/tests/test_http_control.py new file mode 100644 index 000000000..e80523864 --- /dev/null +++ b/tools/openfeature-provider-tck/tests/test_http_control.py @@ -0,0 +1,234 @@ +"""What the Gherkin cannot assert about the HTTP control path. + +Every scenario's isolation rests on :meth:`HttpControl.prepare_scenario` doing +the right thing against a backend that implements only part of the control API, +and on a disconnect being remembered. Both are invisible from inside a scenario: +a control that silently did nothing would leave each scenario running against +whatever state the previous one left behind, and the suite would report those +results as conformance. + +So the control API is stubbed with :mod:`http.server` -- no Docker, no network +beyond loopback -- and the requests it actually made are asserted. +""" + +from __future__ import annotations + +import threading +import typing +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +import pytest + +from openfeature.contrib.tools.provider_tck import ( + ControlApiError, + HttpControl, +) + + +class _StubControlApi: + """A control API that records every request and answers a scripted status.""" + + def __init__(self, statuses: dict[str, int] | None = None) -> None: + self.requests: list[tuple[str, str, str]] = [] + """(method, path, query) of every request, in order.""" + + self.statuses = statuses or {} + stub = self + + class Handler(BaseHTTPRequestHandler): + def do_POST(self) -> None: + path, _, query = self.path.partition("?") + stub.requests.append(("POST", path, query)) + status = stub.statuses.get(path, 200) + body = b'{"status":"stub"}' + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args: typing.Any) -> None: + """Silence the default stderr logging.""" + + self._server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + self._thread = threading.Thread(target=self._server.serve_forever, daemon=True) + + @property + def base_url(self) -> str: + host, port = self._server.server_address[:2] + return f"http://{host!s}:{port}" + + @property + def paths(self) -> list[str]: + return [path for _, path, _ in self.requests] + + def __enter__(self) -> _StubControlApi: + self._thread.start() + return self + + def __exit__(self, *_exc: object) -> None: + self._server.shutdown() + self._server.server_close() + self._thread.join(timeout=5) + + +@pytest.fixture +def stub() -> typing.Iterator[_StubControlApi]: + with _StubControlApi() as api: + yield api + + +def test_prepare_scenario_prefers_reset_when_the_backend_implements_it( + stub: _StubControlApi, +) -> None: + """The preferred primitive, because it causes no availability blip. + + A ``/start`` between scenarios restarts the backend process, which a + provider observes as an outage and may report as a lifecycle event in the + scenario that follows. + """ + control = HttpControl(stub.base_url) + + control.prepare_scenario() + control.prepare_scenario() + + assert stub.paths == ["/reset", "/reset"] + + +def test_prepare_scenario_falls_back_to_start_and_remembers_the_answer() -> None: + """The path flagd-testbed actually takes: its launchpad has no ``/reset``. + + The fallback must be probed once rather than once per scenario -- a wasted + 404 before every scenario is a slow suite, and hiding the probe entirely + would mean a backend that grows ``/reset`` never gets used properly. + """ + with _StubControlApi({"/reset": 404}) as stub: + control = HttpControl(stub.base_url) + + control.prepare_scenario() + control.prepare_scenario() + control.prepare_scenario() + + assert stub.paths == ["/reset", "/start", "/start", "/start"] + + +@pytest.mark.parametrize("status", [404, 501]) +def test_both_documented_not_implemented_statuses_trigger_the_fallback( + status: int, +) -> None: + """The OpenAPI document permits either, so neither may be treated as a failure.""" + with _StubControlApi({"/reset": status}) as stub: + HttpControl(stub.base_url).prepare_scenario() + + assert stub.paths == ["/reset", "/start"] + + +def test_the_scenario_after_a_disconnect_starts_the_backend( + stub: _StubControlApi, +) -> None: + """``/reset`` restores flag state; it is not specified to start a stopped backend. + + Without this the scenario following a disconnect would prepare a backend + that is still down, register a provider against it, and report the failure + as a provider defect. + """ + control = HttpControl(stub.base_url) + control.prepare_scenario() # settles on /reset, which this stub supports + stub.requests.clear() + + control.disconnect() + control.prepare_scenario() + + assert stub.paths == ["/stop", "/start"] + + +def test_reconnect_starts_the_backend_and_clears_the_disconnect( + stub: _StubControlApi, +) -> None: + """A scenario that ended its own outage leaves the backend up, so ``/reset`` is fine again.""" + control = HttpControl(stub.base_url) + control.prepare_scenario() + control.disconnect() + control.reconnect() + stub.requests.clear() + + control.prepare_scenario() + + assert stub.paths == ["/reset"] + + +def test_start_names_the_configuration_under_test() -> None: + """``default`` is the only name every backend must support, and it serves the canonical set.""" + with _StubControlApi({"/reset": 404}) as stub: + HttpControl(stub.base_url).prepare_scenario() + + assert ("POST", "/start", "config=default") in stub.requests + + +def test_a_custom_configuration_is_carried_through() -> None: + with _StubControlApi({"/reset": 404}) as stub: + HttpControl(stub.base_url, configuration="ssl").prepare_scenario() + + assert ("POST", "/start", "config=ssl") in stub.requests + + +def test_restart_carries_the_outage_duration(stub: _StubControlApi) -> None: + HttpControl(stub.base_url).restart(7) + + assert ("POST", "/restart", "seconds=7") in stub.requests + + +def test_change_flag_posts_to_change(stub: _StubControlApi) -> None: + HttpControl(stub.base_url).change_flag() + + assert stub.paths == ["/change"] + + +def test_a_failed_control_call_raises_rather_than_passing_silently() -> None: + """A control call that did nothing would leave the next scenario in an unknown state.""" + with ( + _StubControlApi({"/change": 500}) as stub, + pytest.raises(ControlApiError, match="500"), + ): + HttpControl(stub.base_url).change_flag() + + +def test_an_unreachable_control_api_raises_with_the_reason() -> None: + """The control API must stay up even while the backend is deliberately down.""" + # Bound and immediately closed, so the port is almost certainly free. + with _StubControlApi() as stub: + base_url = stub.base_url + control = HttpControl(base_url, timeout=2.0) + + with pytest.raises(ControlApiError, match="control request POST"): + control.change_flag() + + +@pytest.mark.parametrize( + "base_url", + ["", "localhost:8080", "file:///etc/passwd", "ftp://localhost:8080"], +) +def test_a_base_url_that_is_not_an_http_url_is_rejected_at_construction( + base_url: str, +) -> None: + """Rejected early, and by scheme, so no other URL scheme can reach ``urlopen``.""" + with pytest.raises(ValueError, match="not an http"): + HttpControl(base_url) + + +def test_a_trailing_slash_does_not_produce_a_double_slash_path() -> None: + with _StubControlApi() as stub: + HttpControl(stub.base_url + "/").change_flag() + + assert stub.paths == ["/change"] + + +def test_the_control_reports_which_api_it_drives_the_backend_through() -> None: + """The optional property ``BackendControl`` documents, answered here. + + A control that stays quiet has the field omitted from its report, which puts + the normative HTTP path on the same footing as one that declined to say. This + control can say, so it does. + """ + with _StubControlApi() as stub: + assert HttpControl(stub.base_url).control_api == "http" From 99e7770808063505d538271e4ced9a004c6b5a79 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 18:35:34 +0200 Subject: [PATCH 18/46] ci: run the provider-tck stacked pull requests The pull_request filter matches the BASE branch, so only the suite PR -- the one targeting main -- was ever checked. The report and adoption PRs stacked on it have never run CI, which is why their green ticks meant nothing: the checks on display belong to the base PR. One line, and temporary for the duration of review. The workflow is taken from the head branch, so it has to sit on the base and reach the children by rebase. Signed-off-by: Simon Schrottner --- .github/workflows/build.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index d80adb191..59b8f0df3 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -16,6 +16,13 @@ on: - reopened branches: - main + # Temporary, for the duration of the provider conformance suite's review. + # Without it a stacked pull request gets no CI at all: this filter matches + # the pull request's BASE branch, so only the suite PR itself -- the one + # targeting main -- was ever checked, and the report and adoption PRs + # stacked on it were merged-in-theory and tested never. Remove once the + # chain has landed. See open-feature/spec#417. + - 'feat/provider-tck*' permissions: contents: read From a260d72e0bee9e83bfc5f3f2573bc9ab3f778b18 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 19:02:42 +0200 Subject: [PATCH 19/46] fix(provider-tck): drop not_applicable, a skip already carries its reason "Not declared" and "not applicable" are both skips. Giving them separate representations asks an adopter to learn more vocabulary without telling a reader anything the skip's reason does not already say: the scenario's tags say what was asked, the declaration says whether it was claimed, and the reason says why it was skipped. The gate never distinguished them, and neither did the results payload. The field's own docstring made the argument for removing it. It reserved itself for provider-specific impossibility, on the grounds that an impossibility which is a property of the language belongs in the capability documentation rather than in every report -- and both motivating cases are exactly that. @numeric-coercion cannot hold where the language has a single numeric type; @large-integers cannot hold on a 32-bit accessor. Neither is a fact about a provider, and both are now stated once in Appendix F. Nothing under providers/ ever populated the field. The report schema dropped declaration.notApplicable in open-feature/spec 7f03f672, and Appendix F records where language-level impossibility lives in 600ef9fd. Gone with it: the rule refusing a capability named in both capabilities and not_applicable. It was a rule about holding two claims at once, and with one claim left there is nothing for it to be a rule about -- a capability is declared or it is not, which the dataclass already enforces by having one field. The reserved-capability refusal and the unknown-capability refusal both still apply to what remains. The capabilities docstring and the README now say where a capability that cannot hold in a language at all is recorded, so an adopter who looks for the field finds the answer rather than its absence. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 22 ++--- .../contrib/tools/provider_tck/config.py | 96 ++++--------------- .../tests/test_declaration.py | 34 ------- 3 files changed, 32 insertions(+), 120 deletions(-) diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index 1af1bea08..f79f28141 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -205,10 +205,17 @@ Untagged scenarios are mandatory and always run. `capabilities` defaults to ever capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from the default, run the suite, and remove only what your provider genuinely cannot do. +Leaving a capability out is the only way to withhold it, and one skip carrying its reason is the +whole mechanism: the scenario's tags say what was asked, the declaration says whether it was +claimed, and the skip says why it was not. A capability that cannot hold in a language *at all* — +`@numeric-coercion` where the language has a single numeric type, `@large-integers` on a 32-bit +accessor — is a property of the SDK rather than of the provider, and +[Appendix F][appendix-f] records it once instead of every report restating it. + A reserved capability is documented so the vocabulary has a place for it once scenarios exist, and until then it **must not be declared**. Nothing carries the tag, so declaring it cannot be verified, cannot produce a skip, and tells anyone reading the declaration only that something was claimed and -nothing examined. `TckConfig` raises if you name one in `capabilities` or in `not_applicable`, and +nothing examined. `TckConfig` raises if you name one in `capabilities`, and `DECLARABLE_CAPABILITIES` excludes them — which is the case that matters, because "every capability except X" is how a reserved tag gets declared by accident rather than by decision. One implementation's published conformance report asserts `@targeting` and `@caching` for exactly that @@ -258,15 +265,8 @@ up on and fails its scenario with a message rather than hanging the session. ### Declaring more than a capability set -Two further fields on `TckConfig` say things a capability set cannot, and both are declarations -rather than switches: neither changes which scenarios run or what they assert. - -`not_applicable={Capability.X: "why"}` is for a capability that *cannot* hold rather than one you -chose not to declare. The suite treats the two identically — the scenarios are skipped either way, -with the reason — but collapsing them misrepresents a provider, and whole languages with it: -`@numeric-coercion` is unsatisfiable in JavaScript because the language has no integer type, and -recording that as a choice would show every JavaScript provider as declining something none of them -can have. Declining an optional feature is a choice; an impossibility is not. +One further field on `TckConfig` says something a capability set cannot, and it is a declaration +rather than a switch: it changes neither which scenarios run nor what they assert. `known_deviations=(KnownDeviation(issue=..., summary=...),)` acknowledges a gap against something the specification does *not* treat as optional, with somewhere it is tracked. It is an acknowledgement @@ -408,7 +408,7 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping and the control-API it reports, against a stubbed control API | ``` -125 passed, 21 skipped, 2 xfailed +121 passed, 21 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py index 71127051d..ed0775b7e 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py @@ -3,7 +3,7 @@ from __future__ import annotations import typing -from collections.abc import Callable, Collection, Iterable, Mapping, Sequence +from collections.abc import Callable, Collection, Iterable, Sequence from dataclasses import dataclass, field from openfeature.provider import FeatureProvider @@ -29,10 +29,9 @@ class KnownDeviation: """A gap the provider is known to have, acknowledged rather than hidden. - Distinct from an undeclared capability, which is a choice, and from a - not-applicable one, which is impossible: this is a defect against something - the specification does not treat as optional, with the gap tracked - somewhere. + Distinct from an undeclared capability, which is a choice the provider is + entitled to make: this is a defect against something the specification does + not treat as optional, with the gap tracked somewhere. It changes nothing about how the suite runs. The scenario still fails, and the results payload still reports it as failed -- a report that softened a @@ -136,25 +135,12 @@ class TckConfig: Naming a reserved capability here is rejected at construction rather than passed into a report. See :data:`~.capability.RESERVED_CAPABILITIES`. - """ - - not_applicable: Mapping[Capability, str] = field(default_factory=dict) - """Capabilities that cannot hold for this provider, each with a reason. - - Kept apart from simply leaving a capability out of :attr:`capabilities`, - because the two are different claims and collapsing them misrepresents whole - languages: ``@numeric-coercion`` is unsatisfiable in JavaScript because - the language has no integer type, and reporting that as a choice would show - every JavaScript provider as missing something none of them can have. - Scenarios behind a not-applicable capability are skipped exactly as an - undeclared one's are -- the gate makes no distinction, and neither does the - results payload. The difference is recorded once, here, and reaches the - report's declaration. - - Where the impossibility is a property of the language rather than of the - provider it belongs in the capability documentation rather than in every - report, so this is for provider-specific cases. + A capability that cannot hold in a language at all -- ``@numeric-coercion`` + where the language has a single numeric type, ``@large-integers`` on a + 32-bit accessor -- is a property of the SDK rather than of the provider, and + Appendix F records it once rather than every report restating it. Here it is + simply left undeclared, and the skip carries the reason. """ known_deviations: Sequence[KnownDeviation] = () @@ -216,46 +202,9 @@ def __post_init__(self) -> None: f"the Capability enum" ) - # Normalised the same way, so a dict literal keyed by Capability is what - # an adopter writes and a plain mapping is what everything else reads. - object.__setattr__(self, "not_applicable", dict(self.not_applicable)) object.__setattr__(self, "known_deviations", tuple(self.known_deviations)) - stray = [c for c in self.not_applicable if not isinstance(c, Capability)] - if stray: - problems.append( - f"unknown capabilities {stray!r} in not_applicable: capabilities are " - f"the members of the Capability enum" - ) - - both = sorted( - capability.tag - for capability in self.not_applicable - if isinstance(capability, Capability) and capability in self.capabilities - ) - if both: - problems.append( - f"capabilities and not_applicable both claim {' '.join(both)}: a " - f"capability is either declared or impossible, and a report saying " - f"both leaves a consumer to guess which" - ) - - problems.extend( - reserved_problems(self.capabilities, self.not_applicable.keys()) - ) - - unreasoned = sorted( - capability.tag - for capability, reason in self.not_applicable.items() - if isinstance(capability, Capability) - and (not isinstance(reason, str) or not reason.strip()) - ) - if unreasoned: - problems.append( - f"not_applicable gives no reason for {' '.join(unreasoned)}: " - f"'impossible for this provider' is only useful to a reader who is " - f"told why, and the report schema requires the reason" - ) + problems.extend(reserved_problems(self.capabilities)) if ( Capability.UNAVAILABLE_INIT in self.capabilities @@ -293,14 +242,12 @@ def sorted_capabilities(self) -> list[str]: return sorted(c.tag for c in self.capabilities) -def reserved_problems(*named: Iterable[Capability]) -> list[str]: - """Refuse a reserved capability named anywhere in a configuration. +def reserved_problems(declared: Iterable[Capability]) -> list[str]: + """Refuse a reserved capability named in a configuration. - A reserved capability gates no scenario, so naming it cannot be verified - either way: declaring it claims something nothing examined, and calling it - not-applicable records an impossibility about a question that was never - asked. Either would reach the report's declaration, which the schema - forbids. + A reserved capability gates no scenario, so declaring it cannot be verified + either way: the claim is about something nothing examined, and it would + reach the report's declaration, which the schema forbids. Refused rather than dropped quietly. The adopter wrote it down and meant something by it, so a configuration silently different from the one they @@ -312,19 +259,18 @@ def reserved_problems(*named: Iterable[Capability]) -> list[str]: """ reserved = sorted( capability.tag - for group in named - for capability in group + for capability in declared if isinstance(capability, Capability) and capability.reserved ) if not reserved: return [] declarable = " ".join(sorted(c.tag for c in DECLARABLE_CAPABILITIES)) return [ - f"reserved capabilities {' '.join(sorted(set(reserved)))} cannot be declared " - f"or called not-applicable: no scenario carries them, so the claim cannot be " - f"verified, cannot produce a skip, and would tell a reader of the report only " - f"that something was claimed and nothing examined. The declarable " - f"capabilities, which is what DECLARABLE_CAPABILITIES holds, are {declarable}" + f"reserved capabilities {' '.join(sorted(set(reserved)))} cannot be declared: " + f"no scenario carries them, so the claim cannot be verified, cannot produce a " + f"skip, and would tell a reader of the report only that something was claimed " + f"and nothing examined. The declarable capabilities, which is what " + f"DECLARABLE_CAPABILITIES holds, are {declarable}" ] diff --git a/tools/openfeature-provider-tck/tests/test_declaration.py b/tools/openfeature-provider-tck/tests/test_declaration.py index d6cc3ddac..73dddb84d 100644 --- a/tools/openfeature-provider-tck/tests/test_declaration.py +++ b/tools/openfeature-provider-tck/tests/test_declaration.py @@ -200,8 +200,6 @@ def test_a_reserved_capability_cannot_be_declared() -> None: for reserved in RESERVED_CAPABILITIES: with pytest.raises(ValueError, match=f"reserved capabilities {reserved.tag}"): _config(capabilities={Capability.EVENTS, reserved}) - with pytest.raises(ValueError, match=f"reserved capabilities {reserved.tag}"): - _config(not_applicable={reserved: "no scenario asks"}) def test_the_refusal_says_what_may_be_declared_instead() -> None: @@ -214,38 +212,6 @@ def test_the_refusal_says_what_may_be_declared_instead() -> None: assert capability.tag in message -# -- declaring an impossibility ---------------------------------------------- - - -def test_not_applicable_is_normalised_and_keeps_its_reasons() -> None: - """Written as a dict literal keyed by ``Capability``; read as a mapping.""" - config = _config(not_applicable={Capability.NUMERIC_COERCION: "no integer type"}) - assert dict(config.not_applicable) == { - Capability.NUMERIC_COERCION: "no integer type" - } - - -def test_a_capability_cannot_be_both_declared_and_impossible() -> None: - """The two are different claims, and a declaration asserting both says neither.""" - with pytest.raises(ValueError, match="both claim @events"): - _config( - capabilities={Capability.EVENTS}, - not_applicable={Capability.EVENTS: "a reason"}, - ) - - -def test_a_not_applicable_capability_must_say_why() -> None: - """A reason is required: "impossible for this provider" is useless without one.""" - for empty in ("", " ", "\n"): - with pytest.raises(ValueError, match="no reason for @stale"): - _config(not_applicable={Capability.STALE: empty}) - - -def test_something_that_is_not_a_capability_cannot_be_not_applicable() -> None: - with pytest.raises(ValueError, match="in not_applicable"): - _config(not_applicable={"stale": "a reason"}) - - # -- acknowledging a gap ----------------------------------------------------- From f5e1f727757a3dd397834c9120563b4cb5dc4cd4 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 19:39:37 +0200 Subject: [PATCH 20/46] fix(provider-tck): seed the canonical set from the file, not from a copy canonical_flag_set() hand-wrote the thirteen canonical flags as Python literals. The specification publishes canonical-flags.json so that an adopter can "seed a backend directly from the canonical definition rather than transcribing it, transcription being the usual way the two drift apart" -- and this suite, which packages that very file, transcribed it anyway. The cost was already paid: renaming three flags in the spec meant hand-editing the same set in four languages, and here a second transcription was missed on the first pass. The failure mode is silent. A fixture that has drifted makes the in-memory self-tests pass against a baseline that is no longer the canonical one, so the suite verifies itself against the wrong flags while reporting green, and the report it publishes still claims the canonical set. So decode the packaged file instead. Go, JavaScript and Java already do; this was the last one. Python makes the load-bearing part -- type fidelity -- nearly free, because json.loads gives int for 10, float for 10.0 and an arbitrary-precision int for 2^53 - 1, and the decoder passes a variant's value through untouched. Nearly, not entirely: the self-tests now state those types independently, because normalising integral floats to int is the decoder bug that bit Java and it makes the lossless half of @numeric-coercion pass without coercing anything. $comment is ignored at the document, flag and variant-name levels and deliberately not inside a variant's value: a value is opaque data, and an object flag with a $comment member would be quietly corrupted by a loader that reached into it. JavaScript drew the same line on purpose. The literals are gone rather than kept as a cross-check -- two copies with a test comparing them is the same drift risk with extra steps. What replaces them is a test that every flag the packaged file defines is served under the file's own defaultVariant, read back through the typed resolver a scenario would use. changing-flag stays hand-built, because change_flag has to rebuild it at its other variant and so has to name both; that those two names are the file's is now asserted rather than assumed. canonical_flags_json() moves from __init__ to provider, next to the decoder that consumes it, since the other direction is an import cycle. The public surface is unchanged. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 4 +- .../contrib/tools/provider_tck/__init__.py | 17 +- .../contrib/tools/provider_tck/provider.py | 195 ++++++++----- .../tests/test_in_process_control.py | 258 +++++++++++++++--- 4 files changed, 359 insertions(+), 115 deletions(-) diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index f79f28141..2a6df0a03 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -401,14 +401,14 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | --- | --- | --- | | `test_in_memory_conformance` | the SDK's `InMemoryProvider` | reference adoption for a backend-less provider | | `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | -| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set mirrors `canonical-flags.json` type for type | +| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set is decoded from `canonical-flags.json` — every flag served under the file's own default variant, with the Python type the file wrote | | `test_lifecycle_steps` | the steps that call the provider directly | the in-memory suites skip `@lifecycle`, so the shutdown, re-initialise and metadata steps are driven against a recording provider instead | | `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | | `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity | | `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping and the control-API it reports, against a stubbed control API | ``` -121 passed, 21 skipped, 2 xfailed +140 passed, 21 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py index 20e415d88..18cc4f341 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py @@ -74,6 +74,7 @@ def tck_config(): CHANGING_FLAG_KEY, ControllableInMemoryProvider, canonical_flag_set, + canonical_flags_json, ) from .state import TckState @@ -123,22 +124,6 @@ def tck_config(): _PACKAGE = "openfeature.contrib.tools.provider_tck" -def canonical_flags_json() -> str: - """Return the canonical flag set as raw JSON, in the flagd flag-definition format. - - This is the flag set every scenario assumes, and a backend under test must - serve an equivalent one. The format is not what matters -- the keys, types, - variant names and resolved values are. Seed them however your backend seeds - flags. - - Exposed so an adopting provider can seed a backend from the canonical - definition rather than transcribing it, transcription being the usual way - the two drift apart. - """ - ref = importlib.resources.files(_PACKAGE) / "flag_data" / "canonical-flags.json" - return ref.read_text(encoding="utf-8") - - def control_api_spec() -> str: """Return the OpenAPI document a containerised backend under test must implement.""" ref = importlib.resources.files(_PACKAGE) / "control-api.yaml" diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py index fae7b5640..c674146e3 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py @@ -2,6 +2,8 @@ from __future__ import annotations +import importlib.resources +import json import typing from openfeature.event import ProviderEventDetails @@ -15,6 +17,7 @@ "CHANGING_FLAG_KEY", "ControllableInMemoryProvider", "canonical_flag_set", + "canonical_flags_json", "changing_flag", ] @@ -24,6 +27,23 @@ _CHANGING_BASELINE = "foo" _CHANGING_CHANGED = "bar" +_PACKAGE = "openfeature.contrib.tools.provider_tck" + +_FLAG_DATA_DIRECTORY = "flag_data" +_CANONICAL_FLAGS_FILE = "canonical-flags.json" + +_COMMENT_KEY = "$comment" +"""The key the specification's assets carry prose under. + +Ignored at the document level, at a flag's level and among a flag's *variant +names* -- a "variant" called ``$comment`` is prose about the flag rather than a +variant of it -- and deliberately **not** inside a variant's value. A value is +opaque data the suite passes through: ``object-flag`` could perfectly well grow +a member of that name, and a loader that reached into a value to strip it would +serve an object no scenario expects. JavaScript's suite draws the line in the +same place, on purpose. +""" + class ControllableInMemoryProvider(InMemoryProvider): """An in-memory provider whose flag set can be replaced at runtime. @@ -81,6 +101,13 @@ def flag(self, key: str) -> InMemoryFlag[typing.Any] | None: def changing_flag(default_variant: str) -> InMemoryFlag[str]: + """Build ``changing-flag`` at one of its two variants. + + The one flag built by hand rather than decoded, because + :meth:`InProcessControl.change_flag` has to rebuild it at the *other* + variant and so has to name both. That the names here are the ones the + canonical file defines is asserted by the self-tests rather than assumed. + """ return InMemoryFlag( default_variant=default_variant, variants={ @@ -90,78 +117,124 @@ def changing_flag(default_variant: str) -> InMemoryFlag[str]: ) +def canonical_flags_json() -> str: + """Return the canonical flag set as raw JSON, in the flagd flag-definition format. + + This is the flag set every scenario assumes, and a backend under test must + serve an equivalent one. The format is not what matters -- the keys, types, + variant names and resolved values are. Seed them however your backend seeds + flags. + + Exposed so an adopting provider can seed a backend from the canonical + definition rather than transcribing it, transcription being the usual way + the two drift apart. :func:`canonical_flag_set` takes its own advice. + """ + ref = ( + importlib.resources.files(_PACKAGE) + / _FLAG_DATA_DIRECTORY + / _CANONICAL_FLAGS_FILE + ) + return ref.read_text(encoding="utf-8") + + def canonical_flag_set() -> FlagStorage: """Return the canonical flag set as SDK in-memory flags. - Mirrors ``flag_data/canonical-flags.json`` entry for entry -- and the - self-tests check that it does, value for value and Python type for Python - type. Four properties of that file are load-bearing and hold here too: + Decoded from ``flag_data/canonical-flags.json`` -- the JSON + :func:`canonical_flags_json` returns -- rather than transcribed, so that the + in-memory suites cannot drift from the file every other language seeds a + backend from. That file is published precisely so an adopter can "seed a + backend directly from the canonical definition rather than transcribing it, + transcription being the usual way the two drift apart"; this suite is an + adopter of it like any other. + + The drift it prevents is silent rather than loud. A fixture that has moved + away from the file makes the in-memory self-tests pass against a baseline + that is no longer the canonical one, so the suite verifies itself against + the wrong flags while reporting green -- and the report it publishes claims + the canonical set. + + Four properties of the file are load-bearing, and all four survive the + decoding: * ``missing-flag`` is absent, which is what the ``FLAG_NOT_FOUND`` scenario tests. Adding it turns that scenario green for the wrong reason. * no flag carries a ``context_evaluator``, so every evaluation reports reason ``STATIC`` -- the TCK tests a provider's mapping of a response, not a - backend's evaluation logic. + backend's evaluation logic. Nothing here can add one: the file has no way + to express targeting that this decoder reads. * ``boolean-zero-flag``, ``integer-zero-flag`` and ``string-zero-flag`` resolve to ``False``, ``0`` and ``""``. They are values, not absences, and the falsy scenarios exist to catch a provider that cannot tell the difference. Their ``zero``/``non-zero`` variant names are load-bearing too: the scenarios assert the variant, not only the value. - * ``integral-float-flag`` is the ``float`` ``10.0`` and ``huge-integer-flag`` - is the ``int`` ``9007199254740991``. Writing the first as ``10`` makes the - lossless-coercion scenario pass without coercing; nothing here goes - through a float, so the second cannot be rounded. + * a number keeps the type it was written with. ``json.loads`` gives ``int`` + for ``10``, ``float`` for ``10.0`` and an arbitrary-precision ``int`` for + 2^53 - 1, and nothing here normalises either way, so + ``integral-float-flag`` stays the ``float`` ``10.0`` and + ``huge-integer-flag`` stays exact. Normalising integral floats to ``int`` + is the decoder bug that bit Java, and it makes the lossless-coercion + scenario pass without coercing anything. + + A variant's value is passed through untouched, which is both why the types + survive and why a ``$comment`` member *inside* an object value survives with + them -- see :data:`_COMMENT_KEY`. + + Raises: + ValueError: if the packaged file is not the shape this expects. + Unreachable for a pinned spec revision, because the file is copied + in from the submodule at build time: a failure here means the pinned + assets and this decoder disagree about the file's shape, which + moving the pin should have surfaced. """ - return { - "boolean-flag": InMemoryFlag( - default_variant="on", variants={"on": True, "off": False} - ), - "string-flag": InMemoryFlag( - default_variant="greeting", variants={"greeting": "hi", "parting": "bye"} - ), - "integer-flag": InMemoryFlag( - default_variant="ten", variants={"one": 1, "ten": 10} - ), - "float-flag": InMemoryFlag( - default_variant="half", variants={"tenth": 0.1, "half": 0.5} - ), - # 2^31 - 1: the largest value every language's integer accessor can ask for. - "large-integer-flag": InMemoryFlag( - default_variant="max-int32", variants={"one": 1, "max-int32": 2147483647} - ), - # 2^53 - 1: asked for only under @large-integers. A Python int is exact. - "huge-integer-flag": InMemoryFlag( - default_variant="max-safe", - variants={"one": 1, "max-safe": 9007199254740991}, - ), - # A float with no fractional part, for the lossless half of - # @numeric-coercion. The trailing ``.0`` is the whole point. - "integral-float-flag": InMemoryFlag( - default_variant="ten", variants={"tenth": 0.1, "ten": 10.0} - ), - "boolean-zero-flag": InMemoryFlag( - default_variant="zero", variants={"zero": False, "non-zero": True} - ), - "integer-zero-flag": InMemoryFlag( - default_variant="zero", variants={"zero": 0, "non-zero": 1} - ), - "string-zero-flag": InMemoryFlag( - default_variant="zero", variants={"zero": "", "non-zero": "str"} - ), - "object-flag": InMemoryFlag( - default_variant="template", - variants={ - "empty": {}, - "template": { - "showImages": True, - "title": "Check out these pics!", - "imagesPerPage": 100, - }, - }, - ), - # A string flag, evaluated as a boolean by the TYPE_MISMATCH scenario. - "wrong-flag": InMemoryFlag( - default_variant="one", variants={"one": "uno", "two": "dos"} - ), - CHANGING_FLAG_KEY: changing_flag(_CHANGING_BASELINE), - } + return _decode_canonical_flags(canonical_flags_json()) + + +def _decode_canonical_flags(raw: str) -> FlagStorage: + """Turn the canonical flag file into in-memory flags.""" + document = json.loads(raw) + if not isinstance(document, dict): + msg = f"{_CANONICAL_FLAGS_FILE} is not a JSON object" + raise ValueError(msg) + + # Reading the one member this needs is what ignores $comment at the document + # level, along with every other part of the flagd format the suite has no + # use for. + definitions = document.get("flags") + if not isinstance(definitions, dict) or not definitions: + msg = f"{_CANONICAL_FLAGS_FILE} defines no flags" + raise ValueError(msg) + + return {key: _decode_flag(key, value) for key, value in definitions.items()} + + +def _decode_flag(key: str, definition: typing.Any) -> InMemoryFlag[typing.Any]: + """Turn one flag definition into an in-memory flag, or say why it cannot be.""" + if not isinstance(definition, dict): + msg = f"flag {key!r}: expected an object, got {type(definition).__name__}" + raise ValueError(msg) + + variants = definition.get("variants") + if not isinstance(variants, dict): + msg = f"flag {key!r}: variants is not an object" + raise ValueError(msg) + # Only the variant *names* are filtered. The values are not looked into. + variants = {name: value for name, value in variants.items() if name != _COMMENT_KEY} + + default_variant = definition.get("defaultVariant") + if not isinstance(default_variant, str) or default_variant not in variants: + msg = ( + f"flag {key!r}: default variant {default_variant!r} is not one of its " + f"variants ({', '.join(sorted(map(repr, variants)))})" + ) + raise ValueError(msg) + + raw_state = definition.get("state") + try: + state = InMemoryFlag.State(raw_state) + except ValueError: + allowed = ", ".join(member.value for member in InMemoryFlag.State) + msg = f"flag {key!r}: state {raw_state!r} is none of {allowed}" + raise ValueError(msg) from None + + return InMemoryFlag(default_variant=default_variant, variants=variants, state=state) diff --git a/tools/openfeature-provider-tck/tests/test_in_process_control.py b/tools/openfeature-provider-tck/tests/test_in_process_control.py index ea19864ef..a653bd849 100644 --- a/tools/openfeature-provider-tck/tests/test_in_process_control.py +++ b/tools/openfeature-provider-tck/tests/test_in_process_control.py @@ -8,6 +8,7 @@ import json import typing +from collections.abc import Callable import pytest @@ -19,7 +20,20 @@ canonical_flag_set, canonical_flags_json, ) +from openfeature.contrib.tools.provider_tck.provider import ( + _decode_canonical_flags, + changing_flag, +) +from openfeature.contrib.tools.provider_tck.values import describe, values_equal from openfeature.event import ProviderEvent +from openfeature.flag_evaluation import FlagType, Reason + +_NOT_SEEDED = "this default must never be what a seeded flag resolves to" +"""The default value handed to every resolver below. + +A flag the seeding dropped resolves to it rather than to anything from the file, +which is what the variant and error-code assertions are looking for. +""" def _resolve_changing(provider: ControllableInMemoryProvider) -> str: @@ -125,51 +139,223 @@ def test_canonical_flag_set_omits_missing_flag() -> None: assert "missing-flag" not in canonical_flag_set() -def _same_value_and_type(expected: typing.Any, actual: typing.Any) -> bool: - """Equal, and of the same Python type, member by member. +def _flag_type_of(value: typing.Any) -> FlagType: + """The type a scenario would request a flag of this value as. - ``==`` alone is what a seeding step that "cleans up" gets past: ``10 == 10.0`` - and ``0 == False`` in Python, so the integral float and the falsy values - would compare equal to exactly the mistranslations they exist to catch. + ``bool`` first, because Python makes it a subclass of ``int`` and would + otherwise route ``boolean-zero-flag`` through the integer accessor. """ - if type(expected) is not type(actual): - return False - if isinstance(expected, dict): - return set(expected) == set(actual) and all( - _same_value_and_type(v, actual[k]) for k, v in expected.items() - ) - if isinstance(expected, list): - return len(expected) == len(actual) and all( - _same_value_and_type(e, a) for e, a in zip(expected, actual, strict=True) + if isinstance(value, bool): + return FlagType.BOOLEAN + if isinstance(value, int): + return FlagType.INTEGER + if isinstance(value, float): + return FlagType.FLOAT + if isinstance(value, str): + return FlagType.STRING + return FlagType.OBJECT + + +def test_every_packaged_flag_resolves_to_its_packaged_default_variant() -> None: + """The whole of what seeding from the file has to achieve. + + Every flag the packaged ``canonical-flags.json`` defines is served, under + the variant name the file gives as its ``defaultVariant``, with that + variant's value -- read back through the same typed resolver a scenario + would use, and compared the way the ``Then`` steps compare. + + Asserting the variant and the absence of an error code is what makes this + more than an equality check: a flag the seeding dropped resolves to the + default value with no variant and ``FLAG_NOT_FOUND``, and for the falsy + flags that fallback value can equal what was expected. + """ + canonical = json.loads(canonical_flags_json())["flags"] + provider = ControllableInMemoryProvider(canonical_flag_set()) + + # Annotated explicitly, as in the evaluation step: the five typed resolvers + # have different signatures, so an unannotated mapping infers a value type + # mypy will not let us call. + resolvers: dict[FlagType, Callable[[str, typing.Any], typing.Any]] = { + FlagType.BOOLEAN: provider.resolve_boolean_details, + FlagType.STRING: provider.resolve_string_details, + FlagType.INTEGER: provider.resolve_integer_details, + FlagType.FLOAT: provider.resolve_float_details, + FlagType.OBJECT: provider.resolve_object_details, + } + + assert canonical, "the packaged flag file defines no flags" + for key, definition in canonical.items(): + variant = definition["defaultVariant"] + expected = definition["variants"][variant] + + details = resolvers[_flag_type_of(expected)](key, _NOT_SEEDED) + + assert details.error_code is None, f"{key}: {details.error_message}" + assert details.variant == variant, key + assert details.reason == Reason.STATIC, key + assert values_equal(expected, details.value), ( + f"{key}/{variant}: packaged {describe(expected)}, " + f"resolved {describe(details.value)}" ) - return bool(expected == actual) -def test_canonical_flag_set_mirrors_the_canonical_json_type_for_type() -> None: - """The in-memory flag set is transcribed, so this is what stops it drifting. +# The variants whose Python *type* the scenarios depend on, and what that type +# and value have to be. Not a second copy of the flag set: every key, variant +# and value in it is already checked against the file by the test above, and +# these rows say the one thing a comparison with the file cannot -- that a +# decoder has not normalised a number on its way through. `10 == 10.0` and +# `0 == False` in Python, so the integral float and the falsy values compare +# equal to exactly the mistranslations they exist to catch. +_LOAD_BEARING: tuple[tuple[str, str, type, typing.Any], ...] = ( + ("integer-flag", "ten", int, 10), + ("float-flag", "half", float, 0.5), + ("large-integer-flag", "max-int32", int, 2147483647), + # 2^53 - 1. A Python int is arbitrary-precision, so being an int is being + # exact; arriving as a float would round it. + ("huge-integer-flag", "max-safe", int, 9007199254740991), + # The trailing .0 is the whole point: as an int, the lossless half of + # @numeric-coercion passes without coercing anything. This is the row that + # bit Java. + ("integral-float-flag", "ten", float, 10.0), + ("boolean-zero-flag", "zero", bool, False), + ("integer-zero-flag", "zero", int, 0), + ("string-zero-flag", "zero", str, ""), +) + + +@pytest.mark.parametrize(("key", "variant", "expected_type", "expected"), _LOAD_BEARING) +def test_the_decoded_flag_keeps_the_python_type_the_file_wrote( + key: str, variant: str, expected_type: type, expected: typing.Any +) -> None: + """A number keeps the type it was written with, stated independently of the file.""" + value = canonical_flag_set()[key].variants[variant] + + assert type(value) is expected_type, ( + f"{key}/{variant} decoded to {describe(value)}, expected an " + f"{expected_type.__name__}" + ) + assert value == expected, f"{key}/{variant} decoded to {describe(value)}" + + +def test_a_number_inside_an_object_keeps_its_type_too() -> None: + """A structured flag has to decode the same way on both sides of a comparison. - Key for key, default variant for default variant, and every variant's value - with its Python type: ``json.loads`` keeps ``10.0`` a ``float`` and ``0`` - an ``int``, and the transcription has to as well. The four load-bearing - properties the flag file documents -- no ``missing-flag``, no targeting, - falsy values kept, ``10.0`` a float and 2^53 - 1 an integer -- all follow - from being an exact mirror of it. + ``object-flag``'s expected value reaches the assertion through + ``json.loads`` of the Gherkin table cell. The seeded value reaches it + through ``json.loads`` of the flag file, and nothing converts either, so a + member of the object is the same Python type in both. """ - canonical = json.loads(canonical_flags_json())["flags"] - transcribed = canonical_flag_set() + template = canonical_flag_set()["object-flag"].variants["template"] - assert set(transcribed) == set(canonical) - for key, definition in canonical.items(): - flag = transcribed[key] - assert flag.default_variant == definition["defaultVariant"], key + assert isinstance(template, dict) + assert type(template["imagesPerPage"]) is int, describe(template["imagesPerPage"]) + assert template["imagesPerPage"] == 100 + + +def test_no_packaged_flag_carries_targeting() -> None: + """Every scenario expects reason ``STATIC``. + + The TCK tests a provider's mapping of a response, not a backend's + evaluation logic, so a flag that evaluated its context would report + ``TARGETING_MATCH`` and fail scenarios that are about something else. + """ + for key, flag in canonical_flag_set().items(): assert flag.context_evaluator is None, f"{key} has targeting" - assert set(flag.variants) == set(definition["variants"]), key - for variant, value in definition["variants"].items(): - assert _same_value_and_type(value, flag.variants[variant]), ( - f"{key}/{variant}: canonical {value!r} ({type(value).__name__}), " - f"transcribed {flag.variants[variant]!r} " - f"({type(flag.variants[variant]).__name__})" - ) + + +def test_the_hand_built_changing_flag_matches_the_file() -> None: + """``change_flag`` rebuilds ``changing-flag`` at its other variant. + + That one flag is therefore built by hand rather than decoded, and it names + both variants itself. The file has to define exactly those two, or flipping + between them either changes nothing or invents a variant the backend under + test does not have. + """ + from_file = canonical_flag_set()[CHANGING_FLAG_KEY] + hand_built = changing_flag(from_file.default_variant) + + assert hand_built.variants == from_file.variants + assert from_file.default_variant in hand_built.variants + + +# A document exercising every level a $comment can appear at, including the one +# level it must not be stripped from. +_COMMENTED_DOCUMENT = json.dumps( + { + "$comment": "prose about the document", + "flags": { + "structured-flag": { + "$comment": "prose about the flag", + "state": "ENABLED", + "variants": { + "$comment": "prose about the variants", + "on": {"$comment": "a member of the value, not prose"}, + }, + "defaultVariant": "on", + } + }, + } +) + + +def test_a_comment_is_prose_at_the_document_flag_and_variant_levels() -> None: + """``$comment`` is how the specification's assets carry prose. + + A loader that took one for a flag, or for a variant, would serve a flag + nothing asked for and offer a variant no scenario can resolve. + """ + flags = _decode_canonical_flags(_COMMENTED_DOCUMENT) + + assert set(flags) == {"structured-flag"} + assert set(flags["structured-flag"].variants) == {"on"} + + +def test_a_comment_inside_a_variant_value_is_part_of_the_value() -> None: + """The line the other languages drew deliberately. + + A variant's value is opaque data. An object flag with a ``$comment`` member + is a perfectly good object flag, and a loader that reached into the value to + strip it would serve an object no scenario expects -- silently, because the + rest of the object still matches. + """ + value = _decode_canonical_flags(_COMMENTED_DOCUMENT)["structured-flag"].variants[ + "on" + ] + + assert value == {"$comment": "a member of the value, not prose"} + + +@pytest.mark.parametrize( + ("document", "message"), + [ + ("[]", "not a JSON object"), + ('{"flags": {}}', "defines no flags"), + ('{"flags": {"a": []}}', "expected an object"), + ('{"flags": {"a": {"state": "ENABLED"}}}', "variants is not an object"), + ( + '{"flags": {"a": {"state": "ENABLED", "variants": {"on": 1}, ' + '"defaultVariant": "off"}}}', + "is not one of its variants", + ), + ( + '{"flags": {"a": {"state": "PARTLY", "variants": {"on": 1}, ' + '"defaultVariant": "on"}}}', + "is none of", + ), + ], +) +def test_a_flag_file_this_decoder_does_not_understand_is_refused( + document: str, message: str +) -> None: + """Unreachable for a pinned spec revision, and it says which flag if it happens. + + The assets are copied in from the submodule at build time, so a failure here + means the pinned assets and this decoder disagree about the file's shape -- + which moving the pin should have surfaced. Refusing beats seeding a flag set + that is quietly missing a flag. + """ + with pytest.raises(ValueError, match=message): + _decode_canonical_flags(document) def test_update_flags_names_the_union_of_old_and_new_keys() -> None: From c58d07e99e5502aeaa1c06917bdb03edeca49013 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 23:30:36 +0200 Subject: [PATCH 21/46] feat(provider-tck): gate the variant, and make @targeting real Follows spec 26362f85. The pin and the code move together, because the suite reads its Gherkin from the pin. @variants is new, and it is the one capability found the hard way. Every evaluation scenario asserted a variant, which reads as obviously correct until a backend with no variant concept for a plain flag is put under test: its response carries no such key, the provider never receives one, and no seeding can produce one. Ten scenarios failed a conformant provider for something its author could not fix, and nothing could be recorded as a KnownDeviation because there was no capability to hang one on. Requirement 2.2.4 is a SHOULD and types.md types the field "variant (string, optional)", so the suite was asserting a MUST neither of them states. The assertions now live in one gated Scenario Outline of eight rows; value and reason stay untagged, because 2.2.3 makes the value a MUST. @targeting stops being reserved. The scope argument that reserved it still holds -- its three scenarios do not test how a backend evaluates a rule -- but the conclusion did not: they show the context reached the backend at all, which is a property of the provider and of nothing else. targeting-key-flag carries the one rule in the canonical set, so a matching context resolving to a different value catches a provider that drops the context, with no echo endpoint on the control API. The refusal test asked about @targeting by name, which is how it went stale; it now takes whatever RESERVED_CAPABILITIES holds. The new step wording is Appendix B's, which the flagd testkit here already carries a definition for, and the evaluation context is threaded through the resolve call positionally -- None included, so a scenario that declared no context sends what a two-argument call would. Until the evaluation-context scenarios landed, no scenario supplied a context at all, so a provider that threw on any context passed the whole suite. Neither in-memory suite declares @targeting: _decode_canonical_flags reads only state, variants and defaultVariant, so the rule is inert and targeting-key-flag is served at its miss default. Decoding a rule language would make this package a second implementation of somebody else's evaluator. Both declare @variants, an in-memory flag set being keyed by variant name. 52 canonical scenario instances, up from 40. 158 passed, 27 skipped, 2 xfailed. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 50 +++++++++++-- tools/openfeature-provider-tck/spec | 2 +- .../contrib/tools/provider_tck/capability.py | 71 +++++++++++++++++-- .../contrib/tools/provider_tck/provider.py | 16 +++-- .../contrib/tools/provider_tck/state.py | 11 +++ .../tools/provider_tck/steps/flag_steps.py | 51 ++++++++++++- .../tests/test_controllable_conformance.py | 5 ++ .../tests/test_declaration.py | 26 +++++-- .../tests/test_in_memory_conformance.py | 18 ++++- .../tests/test_in_process_control.py | 20 +++++- 10 files changed, 237 insertions(+), 33 deletions(-) diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index 2a6df0a03..0c2d7ffb3 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -150,7 +150,7 @@ suite at all, so `pytest.skip` carries the reason into the report: ``` SKIPPED provider does not declare capability @stale. - Declared: @events @large-integers @object + Declared: @events @large-integers @object @variants ``` | Capability | Tag | Meaning | @@ -160,11 +160,12 @@ SKIPPED provider does not declare capability @stale. | `Capability.STALE` | `@stale` | enters `STALE` and emits `PROVIDER_STALE` on backend loss | | `Capability.CONFIGURATION_CHANGE` | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | | `Capability.OBJECT` | `@object` | supports structured flag values | +| `Capability.VARIANTS` | `@variants` | names the variant it resolved, which [Requirement 2.2.4](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) makes a `SHOULD` and `types.md` types as optional | | `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | | `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | | `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `Capability.REINITIALIZATION` | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) permits rather than requires | -| `Capability.TARGETING` | `@targeting` | reserved; **not declarable** — no scenarios yet | +| `Capability.TARGETING` | `@targeting` | resolves a flag differently for a matching evaluation context | | `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | `@lifecycle` and `@events` are deliberately separate, and the split matters in both directions. An @@ -201,6 +202,30 @@ alone leaves the scenario skipped on `@lifecycle` and the declaration unverified withholds `LIFECYCLE` has never run this scenario, and has no evidence either way on which to declare reuse. +`@variants` was the one found the hard way, and it is the reason the rule above is worth stating +twice. Every evaluation scenario used to assert a variant, which reads as obviously correct until a +backend with no variant concept for a plain flag is put under test: its evaluation response carries +no such key, the provider never receives one, and no seeding can produce one. Ten scenarios failed a +conformant provider for something its author could not fix, and nothing could be recorded as a +`KnownDeviation` because there was no capability to hang one on. +[Requirement 2.2.4](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) +is a **SHOULD** and `types.md` types the field `variant (string, optional)`, so the suite was +asserting a `MUST` neither of them states. Since spec revision `26362f85` the variant assertions +live in one gated Scenario Outline of eight rows; the value and reason assertions stay untagged, +because 2.2.3 makes the value a `MUST`. + +`@targeting` was **reserved and undeclarable** until the same revision, on the reading that targeting +is backend evaluation logic and out of scope. The scope argument still holds — its three scenarios do +not test how a backend evaluates a rule — but the conclusion did not: they exist to show the context +reached the backend at all, which is a property of the provider and of nothing else. +`targeting-key-flag` is the one flag in the canonical set with a rule, specified by behaviour rather +than syntax (resolve `hit` when the targeting key is exactly `5c3d8535-f81a-4478-a6d3-afaa4d51199e`), +and a matching context resolving to a different value is what catches a provider that drops the +context — no echo endpoint on the control API required. The three scenarios are the matching context, +the non-matching one and no context at all; the second and third are not padding, since a provider +that always returned the targeted value would pass the first and one that refuses to evaluate a rule +with no targeting key is caught by the third. + Untagged scenarios are mandatory and always run. `capabilities` defaults to every *declarable* capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from the default, run the suite, and remove only what your provider genuinely cannot do. @@ -219,7 +244,11 @@ nothing examined. `TckConfig` raises if you name one in `capabilities`, and `DECLARABLE_CAPABILITIES` excludes them — which is the case that matters, because "every capability except X" is how a reserved tag gets declared by accident rather than by decision. One implementation's published conformance report asserts `@targeting` and `@caching` for exactly that -reason. +reason, back when both were reserved. + +`@caching` is the only reserved tag left. Leaving one reserved once it *has* scenarios would be the +mirror of the mistake the set exists to prevent — a capability that can be verified, refused the +chance — so `@targeting` moved out of it the moment the specification gave it three. `@numeric-coercion` deserves a note, because it is the one capability here that **the specification does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of @@ -408,7 +437,7 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping and the control-API it reports, against a stubbed control API | ``` -140 passed, 21 skipped, 2 xfailed +158 passed, 27 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; @@ -419,12 +448,19 @@ Neither in-memory suite declares `@lifecycle`, so the six lifecycle scenarios initialisation, three about shutdown — are skipped in both. That is the point: with no backend to reach, the initialisation ones would pass without testing anything — which is what they did while the feature was gated on `@events`. Neither declares `@numeric-coercion` either, for the reason in -finding 3, so its three scenarios are skipped too. +finding 3, so its three scenarios are skipped too. Neither declares `@targeting`: both resolve the +same decoded flag set, and `canonical_flag_set` deliberately ignores `targeting-key-flag`'s rule +rather than becoming a second implementation of somebody else's evaluator, so those three scenarios +are skipped as well. Both declare `@variants`, since an in-memory flag set is keyed by variant name. ## Known gaps -- **Evaluation context passthrough is unverifiable.** The scenarios build evaluation contexts but - cannot assert one *reached* the backend. That needs an echo operation on the control API. +- **Evaluation context passthrough is verified only for the targeting key.** `targeting-key-flag` + resolves differently for a matching context, so the `@targeting` scenarios catch a provider that + drops the context — no echo operation needed for that. What is still unverified is that the + *whole* context arrives intact: a provider that forwards the targeting key and silently discards + every other attribute passes. That needs either an echo operation on the control API or a second + canonical flag whose rule keys on a custom attribute. - **No shared containerised-backend helper.** `HttpControl` drives the control API, but starting the stack and discovering its mapped ports is still each adopter's own code. Abstracting that from a single example tends to produce the wrong abstraction; it should wait for a second adopter. diff --git a/tools/openfeature-provider-tck/spec b/tools/openfeature-provider-tck/spec index fc99d5ace..26362f85b 160000 --- a/tools/openfeature-provider-tck/spec +++ b/tools/openfeature-provider-tck/spec @@ -1 +1 @@ -Subproject commit fc99d5ace4da472a5fea0595fa4db8034bbbc769 +Subproject commit 26362f85b7fcd59b35b969e6feebee80e206b24f diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py index 69beb1953..13aaa3186 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py @@ -58,6 +58,34 @@ class Capability(str, Enum): OBJECT = "object" """Provider supports structured (object) flag values.""" + VARIANTS = "variants" + """Provider names the variant it resolved. + + Gated because a variant is optional rather than required. `Requirement 2.2.4 + `_ + is a **SHOULD** -- in normal execution a provider "SHOULD populate the + resolution details structure's variant field" -- and ``types.md`` types the + field ``variant (string, optional)``. The same section adds that the value + "might only be meaningful in the context of the flag management system + associated with the provider". + + Some backends have no variant concept for a plain flag at all. Their + evaluation response carries no such key, so the provider never receives one + and no amount of seeding can produce one. Asserting a variant in every + evaluation scenario failed such a backend ten times over for something that + is not a defect and that no provider author can fix -- and left nothing to + record as a :class:`~.config.KnownDeviation`, because there was no + capability to hang one on. + + Declaring it runs one Scenario Outline that asserts the variant for each of + the eight flags whose variant name the canonical set fixes. Withholding it + skips those rows with the reason and changes nothing else: the value and + reason assertions live in untagged scenarios, because + `Requirement 2.2.3 + `_ + makes the value a **MUST**. + """ + UNAVAILABLE_INIT = "unavailable" """Provider reports an error state promptly against a backend it cannot reach.""" @@ -159,8 +187,35 @@ class Capability(str, Enum): """ TARGETING = "targeting" - """Reserved, and **not declarable**. No scenario carries this tag: targeting - is backend evaluation logic.""" + """Provider resolves a flag differently for a matching evaluation context. + + Reserved and undeclarable until spec revision ``26362f85``, on the reading + that targeting is backend evaluation logic and therefore out of scope. The + scope argument still holds -- what the three scenarios test is not how a + backend evaluates a rule -- but the conclusion did not: they exist to show + that the **context reached the backend at all**, which is a property of the + provider and of nothing else. + + ``targeting-key-flag`` is the one flag in the canonical set with a rule, and + it is what makes passthrough observable without an echo endpoint on the + control API: a matching context resolves ``hit`` where anything else + resolves ``miss``, so a provider that drops the context on the floor is + caught by the resolved value itself. The rule is specified by behaviour + rather than by syntax -- resolve ``hit`` when the targeting key is exactly + ``5c3d8535-f81a-4478-a6d3-afaa4d51199e`` -- so a backend expresses it + however it expresses targeting. + + The three scenarios are the matching context, the non-matching one and no + context at all. The second and third are not padding: a provider that always + returned the targeted value would pass the first, and one that refuses to + evaluate a rule with no targeting key present is caught by the third. + + Declare it if the backend under test can express that rule and the provider + forwards the targeting key. A backend with no targeting at all leaves it + undeclared and the three scenarios are skipped with the reason -- which is + also the right answer for an in-memory flag set whose decoder ignores the + ``targeting`` member, as this package's own does. + """ CACHING = "caching" """Reserved, and **not declarable**. No scenario carries this tag yet.""" @@ -179,9 +234,7 @@ def __str__(self) -> str: return self.tag -RESERVED_CAPABILITIES: frozenset[Capability] = frozenset( - {Capability.TARGETING, Capability.CACHING} -) +RESERVED_CAPABILITIES: frozenset[Capability] = frozenset({Capability.CACHING}) """Capabilities that exist in the vocabulary and gate no scenario. They are documented so the vocabulary has a place for them when scenarios exist, @@ -194,6 +247,11 @@ def __str__(self) -> str: :data:`DECLARABLE_CAPABILITIES`, by :attr:`Capability.reserved` and by the validation in :class:`~.config.TckConfig` -- so that the set and the rule cannot drift apart. + +``@caching`` is the only one left. :attr:`Capability.TARGETING` was here until +spec revision ``26362f85`` gave it scenarios, and leaving a tag reserved once it +has them would be the mirror of the mistake this set exists to prevent: a +capability that *can* be verified and is refused the chance. """ DECLARABLE_CAPABILITIES: frozenset[Capability] = ( @@ -210,7 +268,8 @@ def __str__(self) -> str: convenience is exactly how a reserved tag reaches a report by accident: an adopter writing "every capability except X" picks up every reserved tag on the way past, which is how one implementation came to report ``@targeting`` and -``@caching`` as declared without anyone deciding to claim them. +``@caching`` as declared without anyone deciding to claim them -- back when both +were reserved. """ _BY_MARKER: dict[str, Capability] = {c.value: c for c in Capability} diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py index c674146e3..a0c273aa4 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py @@ -160,14 +160,22 @@ def canonical_flag_set() -> FlagStorage: * ``missing-flag`` is absent, which is what the ``FLAG_NOT_FOUND`` scenario tests. Adding it turns that scenario green for the wrong reason. * no flag carries a ``context_evaluator``, so every evaluation reports reason - ``STATIC`` -- the TCK tests a provider's mapping of a response, not a - backend's evaluation logic. Nothing here can add one: the file has no way - to express targeting that this decoder reads. + ``STATIC``. ``targeting-key-flag`` is the one flag in the file with a + ``targeting`` member, and this decoder reads only ``state``, ``variants`` + and ``defaultVariant`` -- so that flag is served at its ``miss`` default + whatever the context, like every other. That is deliberate rather than + pending: decoding a rule language would make this package a second + implementation of somebody else's evaluator, and the untargeted scenarios + are the ones it exists to serve. The consequence is that an in-memory + adoption must leave :attr:`~.capability.Capability.TARGETING` undeclared, + and its three scenarios are skipped with that reason. * ``boolean-zero-flag``, ``integer-zero-flag`` and ``string-zero-flag`` resolve to ``False``, ``0`` and ``""``. They are values, not absences, and the falsy scenarios exist to catch a provider that cannot tell the difference. Their ``zero``/``non-zero`` variant names are load-bearing - too: the scenarios assert the variant, not only the value. + too, for an adoption declaring + :attr:`~.capability.Capability.VARIANTS`: the gated variant scenario + asserts the variant, where the falsy scenarios assert only the value. * a number keeps the type it was written with. ``json.loads`` gives ``int`` for ``10``, ``float`` for ``10.0`` and an arbitrary-precision ``int`` for 2^53 - 1, and nothing here normalises either way, so diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py index 074005f59..694858ee2 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py @@ -13,6 +13,7 @@ from dataclasses import dataclass, field from openfeature.client import OpenFeatureClient +from openfeature.evaluation_context import EvaluationContext from openfeature.event import EventDetails, ProviderEvent from openfeature.flag_evaluation import FlagType from openfeature.provider import FeatureProvider @@ -129,6 +130,16 @@ class TckState: flag_key: str | None = None flag_type: FlagType | None = None default_value: typing.Any = None + evaluation_context: EvaluationContext | None = None + """The context the scenario supplies to the evaluation, if it supplies one. + + ``None`` rather than an empty context, and the distinction is load-bearing: + one of the ``@targeting`` scenarios is specifically about a rule that cannot + match because no context was given at all, and a provider that would fall + over on an empty context rather than on an absent one is exactly what it is + looking for. So an unset context is passed to the SDK as ``None``, which is + what an application calling the two-argument form sends. + """ last: EvaluationRecord | None = None lifecycle: list[LifecycleRecord] = field(default_factory=list) """Every direct lifecycle call this scenario made, in order.""" diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py index 812c5047b..98ecf0de7 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py @@ -7,12 +7,15 @@ from pytest_bdd import given, parsers, then, when +from openfeature.evaluation_context import EvaluationContext from openfeature.flag_evaluation import FlagType +from ..capability import Capability from ..state import EvaluationRecord, TckState from ..values import describe, parse_flag_type, parse_value, values_equal __all__ = [ + "a_context_containing_a_targeting_key", "a_flag_with_key_and_default", "no_exception_should_have_been_thrown", "the_error_code_should_be", @@ -48,6 +51,30 @@ def a_flag_with_key_and_default( tck_state.default_value = parse_value(parsed_type, default) +@given( + parsers.re(r'^a context containing a targeting key with value "(?P[^"]*)"$') +) +def a_context_containing_a_targeting_key(tck_state: TckState, value: str) -> None: + """Supply the evaluation context the resolve call is made with. + + The wording is `Appendix B + `_'s, + which the flagd testkit in this repository already carries a step definition + for, because a second way to say "a context containing a targeting key" is + the divergence Appendix F exists to prevent. + + Only the targeting key, for now. ``targeting-key-flag``'s rule keys on it, + so it is what the canonical set can observe arriving; a custom attribute + would need a second flag whose rule keys on one, which Appendix F lists as + a known gap. + """ + context = tck_state.evaluation_context + if context is None: + tck_state.evaluation_context = EvaluationContext(targeting_key=value) + else: + context.targeting_key = value + + @when("the flag was evaluated with details") def the_flag_was_evaluated_with_details(tck_state: TckState) -> None: """Resolve the declared flag through the typed client call matching its type.""" @@ -56,7 +83,9 @@ def the_flag_was_evaluated_with_details(tck_state: TckState) -> None: # Annotated explicitly: the five typed getters have different signatures, so # an unannotated mapping infers a value type mypy will not let us call. - calls: dict[FlagType, Callable[[str, typing.Any], typing.Any]] = { + calls: dict[ + FlagType, Callable[[str, typing.Any, EvaluationContext | None], typing.Any] + ] = { FlagType.BOOLEAN: client.get_boolean_details, FlagType.STRING: client.get_string_details, FlagType.INTEGER: client.get_integer_details, @@ -66,7 +95,14 @@ def the_flag_was_evaluated_with_details(tck_state: TckState) -> None: record = EvaluationRecord() try: - details = calls[flag_type](key, default) + # Passed positionally and unconditionally, ``None`` included: the SDK's + # own default for the parameter is ``None``, so a scenario that declared + # no context sends what a two-argument call would. Requirement 2.2.1 + # makes the context a parameter of every resolve method, and until the + # evaluation-context scenarios landed no scenario supplied one -- so a + # provider that threw on any context, or serialised it into a malformed + # request, passed the whole suite. + details = calls[flag_type](key, default, tck_state.evaluation_context) except BaseException as exc: # recorded here, asserted on by its own step record.raised = exc record.value = default @@ -103,12 +139,21 @@ def the_resolved_value_should_be(tck_state: TckState, expected: str) -> None: @then(parsers.re(r'^the variant should be "(?P[^"]*)"$')) def the_variant_should_be(tck_state: TckState, expected: str) -> None: + """Assert the resolved variant, for an adoption that declared ``@variants``. + + Reached only through the gated scenario, so the failure message says which + of the two readings applies: a variant lost in transit is a defect, while a + backend with no variant concept at all should withhold the capability rather + than fail here. Requirement 2.2.4 is a ``SHOULD``. + """ record = tck_state.require_evaluation() if record.variant != expected: msg = ( f"variant was {record.variant!r}, expected {expected!r}. A variant that " f"does not survive the trip from the backend is one of the easiest parts " - f"of the contract to drop" + f"of the contract to drop -- but if this backend has no variant concept " + f"for a plain flag, withhold {Capability.VARIANTS.tag} instead: " + f"requirement 2.2.4 is a SHOULD and types.md types the field as optional" ) raise AssertionError(msg) diff --git a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py index 1d36dec21..5155ac803 100644 --- a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py @@ -44,6 +44,10 @@ def tck_config() -> TckConfig: inherits the SDK provider's refusal to coerce: ``10.0`` requested as an integer is a ``TYPE_MISMATCH`` rather than ``10``. ``LARGE_INTEGERS`` is declared, since a Python ``int`` is exact at 2^53 - 1. + + ``TARGETING`` stays undeclared for the reason given there as well: this is + the same decoded flag set, and it ignores ``targeting-key-flag``'s rule. + ``VARIANTS`` is declared, since the flag set is keyed by variant name. """ control = InProcessControl() return TckConfig( @@ -54,6 +58,7 @@ def tck_config() -> TckConfig: Capability.EVENTS, Capability.CONFIGURATION_CHANGE, Capability.OBJECT, + Capability.VARIANTS, Capability.LARGE_INTEGERS, }, ) diff --git a/tools/openfeature-provider-tck/tests/test_declaration.py b/tools/openfeature-provider-tck/tests/test_declaration.py index 73dddb84d..9c4b19200 100644 --- a/tools/openfeature-provider-tck/tests/test_declaration.py +++ b/tools/openfeature-provider-tck/tests/test_declaration.py @@ -110,9 +110,15 @@ def test_a_reserved_capability_is_one_no_canonical_scenario_carries() -> None: """The fact the rule rests on, read off the assets rather than asserted. "Reserved" claims that nothing carries the tag, so declaring it cannot be - verified. If the specification adds a scenario for ``@targeting``, that - stops being true and this fails -- which is the moment the capability should - become declarable, and the moment somebody has to notice. + verified. When the specification gives a reserved tag a scenario, that stops + being true and this fails -- which is the moment the capability should become + declarable, and the moment somebody has to notice. It has already happened + once: ``@targeting`` gained three scenarios at spec revision ``26362f85`` + and moved out of the reserved set, which is what this half is for. + + The other half is the converse, and it is the half that catches a tag added + to the enum and never wired to anything: every declarable capability must be + carried by some canonical scenario, or declaring it examines nothing. """ carried = _canonical_tags() for capability in RESERVED_CAPABILITIES: @@ -154,7 +160,7 @@ def test_the_default_is_every_declarable_capability() -> None: "Declare everything, then narrow it" is the advice, which makes the default the one place a reserved tag would otherwise get declared by accident. One implementation's published report asserts ``@targeting`` and ``@caching`` - for precisely that reason. + for precisely that reason -- back when both were reserved. """ # Not routed through ``_config``, which narrows the set: the field default is # the whole point of this one. It needs an unavailable-provider factory @@ -203,9 +209,17 @@ def test_a_reserved_capability_cannot_be_declared() -> None: def test_the_refusal_says_what_may_be_declared_instead() -> None: - """A message that names the rule and not just the violation.""" + """A message that names the rule and not just the violation. + + The offending capability is taken from ``RESERVED_CAPABILITIES`` rather than + named, because naming one is how this test went stale: it asked about + ``@targeting``, which stopped being reserved the moment the specification + gave it scenarios, and the refusal it was asserting became correct + behaviour's absence. + """ + reserved = next(iter(sorted(RESERVED_CAPABILITIES, key=lambda c: c.tag))) with pytest.raises(ValueError) as raised: - _config(capabilities={Capability.TARGETING}) + _config(capabilities={reserved}) message = str(raised.value) assert "DECLARABLE_CAPABILITIES" in message for capability in DECLARABLE_CAPABILITIES: diff --git a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py index 18fef5c05..b5a6544e2 100644 --- a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py @@ -83,8 +83,17 @@ def tck_config() -> TckConfig: ``ConnectionControl`` for the same reason, and the two omissions keep each other honest: the scenarios are skipped before any step can reach an operation the control cannot perform. - * ``TARGETING`` and ``CACHING`` -- omitted because no scenario carries their - tags yet, so leaving them out skips nothing. + * ``TARGETING`` -- omitted because this flag set has no targeting rule to + resolve. ``canonical-flags.json`` gives ``targeting-key-flag`` one, and + ``_decode_canonical_flags`` deliberately ignores the member: decoding a + rule language would make this package a second implementation of somebody + else's evaluator. So the flag is served at its ``miss`` default whatever + the context, the matching-context scenario would fail, and withholding + the capability is the honest report. That is a property of this in-memory + flag set rather than a defect in the SDK's provider, which is why nothing + here is a ``KnownDeviation``. + * ``CACHING`` -- omitted because no scenario carries the tag yet, so leaving + it out skips nothing. It is also reserved, so declaring it is refused. * ``LIFECYCLE`` -- omitted because there is no backend to reach. The capability asserts that initialisation actually contacts a backend and that the outcome is observable; this provider's ``initialize`` is a no-op @@ -105,7 +114,9 @@ def tck_config() -> TckConfig: rather than a deviation. ``LARGE_INTEGERS`` is declared: a Python ``int`` is unbounded and nothing - in this provider routes a value through a float. + in this provider routes a value through a float. ``VARIANTS`` is declared + too: the in-memory flag set is keyed by variant name, so the provider has + one to report for every flag and does. """ return TckConfig( name="in-memory", @@ -114,6 +125,7 @@ def tck_config() -> TckConfig: capabilities={ Capability.EVENTS, Capability.OBJECT, + Capability.VARIANTS, Capability.LARGE_INTEGERS, }, ) diff --git a/tools/openfeature-provider-tck/tests/test_in_process_control.py b/tools/openfeature-provider-tck/tests/test_in_process_control.py index a653bd849..7545ee4af 100644 --- a/tools/openfeature-provider-tck/tests/test_in_process_control.py +++ b/tools/openfeature-provider-tck/tests/test_in_process_control.py @@ -252,14 +252,28 @@ def test_a_number_inside_an_object_keeps_its_type_too() -> None: assert template["imagesPerPage"] == 100 -def test_no_packaged_flag_carries_targeting() -> None: - """Every scenario expects reason ``STATIC``. +def test_no_decoded_flag_carries_targeting() -> None: + """Every untargeted scenario expects reason ``STATIC``. The TCK tests a provider's mapping of a response, not a backend's evaluation logic, so a flag that evaluated its context would report ``TARGETING_MATCH`` and fail scenarios that are about something else. + + Stated about the *decoded* flags rather than about the file, because since + spec revision ``26362f85`` the file is not free of targeting: it gives + ``targeting-key-flag`` a rule so that a real backend can show a context + arriving. ``_decode_canonical_flags`` reads only ``state``, ``variants`` and + ``defaultVariant``, so the member is inert here -- which is what this + asserts, and what obliges an in-memory adoption to leave ``@targeting`` + undeclared rather than fail its scenarios. """ - for key, flag in canonical_flag_set().items(): + decoded = canonical_flag_set() + assert "targeting-key-flag" in decoded, ( + "the packaged flag file no longer defines targeting-key-flag, so this " + "test no longer checks anything: the flag with the one targeting rule " + "is what makes the member's inertness observable" + ) + for key, flag in decoded.items(): assert flag.context_evaluator is None, f"{key} has targeting" From b6368f439d7ca2b1a62079aac9e79dec9cf479e6 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 12:23:31 +0200 Subject: [PATCH 22/46] feat(provider-tck): add the @disabled-flags capability Follows spec 009afe06. The pin and the code move together, because the suite reads its Gherkin from the pin. The canonical set gains four flags -- disabled-boolean-flag, disabled-string-flag, disabled-integer-flag and disabled-float-flag -- mirroring boolean-flag, string-flag, integer-flag and float-flag exactly, differing only in state. One Scenario Outline of four rows asserts that each resolves to the caller's default. 18 canonical flags, up from 14. Gated because it needs two things and only one comes for free. The caller's default is held by the provider, which always has it; what the provider also needs is a signal that the flag was disabled, told apart from an ordinary resolution and from a missing flag, and that belongs to the backend and its protocol. One with no disabled state, or one answering FLAG_NOT_FOUND for a disabled flag, gives the provider nothing to act on. Appendix F draws the line elsewhere, and the runs behind this commit do not bear that out. It has it that a provider whose backend decides, "such as one speaking OFREP, cannot: the server never sees the caller's default, so it has no way to return it". flagd's RPC resolver is a remote evaluator by exactly that description and satisfies the capability, on the strength of a response carrying reason DISABLED with no variant and no value. flagd's OFREP endpoint answers the same flag with {"reason": "DISABLED"} and no value and no variant -- the same signal in another envelope -- and the Python OFREP provider already substitutes the caller's default for the absent value; it fails these scenarios for an unrelated reason its own suite records. The discrepancy belongs upstream rather than papered over here. What it changes locally is only what a withheld declaration may be read as: not necessarily an impossibility, so read the adoption's note for which it was. Nothing in the specification says what a provider owes a disabled flag. Requirement 1.4.7 is about the SDK propagating whatever reason arrived, and 2.2.5 only lists DISABLED among the reason strings a provider may use. So Appendix F states the behaviour, the way it does for @numeric-coercion, and gates it. The rows assert the value and the absence of an error, not the reason -- pinning DISABLED would rest on 2.2.5's SHOULD and its "some other string" -- and not the variant, since a disabled flag has resolved none: @disabled-flags and @variants deliberately do not compose. Neither in-memory suite declares it, which is finding 4. InMemoryFlag has a State enum, takes one in its constructor and never reads it: resolve() returns the default variant whatever the state. Measured before the tag was gated, all four rows failed on the value in both suites, disabled-boolean-flag resolving to True against a caller default of false. _decode_canonical_flags is not where it stops -- it reads the file's state, validates it and passes it through faithfully -- so the self-tests now pin that the state reaches exactly those four flags and no others, and that each still mirrors its enabled counterpart, since two variants that resolved alike would make a row vacuous. The four flags stay in the sweep that resolves every packaged flag through its typed resolver rather than being excluded from it. They pass there, because this provider cannot tell a disabled flag from an enabled one, and a failure now says so and says that the capability has become declarable. 56 canonical scenario instances, up from 52. 163 passed, 35 skipped, 2 xfailed. Signed-off-by: Simon Schrottner --- tools/openfeature-provider-tck/README.md | 64 +++++++++- tools/openfeature-provider-tck/spec | 2 +- .../contrib/tools/provider_tck/capability.py | 69 +++++++++++ .../tests/test_controllable_conformance.py | 6 + .../tests/test_in_memory_conformance.py | 13 +++ .../tests/test_in_process_control.py | 109 +++++++++++++++++- 6 files changed, 254 insertions(+), 9 deletions(-) diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-provider-tck/README.md index 0c2d7ffb3..6e8ff5337 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-provider-tck/README.md @@ -161,6 +161,7 @@ SKIPPED provider does not declare capability @stale. | `Capability.CONFIGURATION_CHANGE` | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | | `Capability.OBJECT` | `@object` | supports structured flag values | | `Capability.VARIANTS` | `@variants` | names the variant it resolved, which [Requirement 2.2.4](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) makes a `SHOULD` and `types.md` types as optional | +| `Capability.DISABLED_FLAGS` | `@disabled-flags` | resolves a flag disabled in the management system to the code default | | `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | | `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | | `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | @@ -226,6 +227,39 @@ the non-matching one and no context at all; the second and third are not padding that always returned the targeted value would pass the first and one that refuses to evaluate a rule with no targeting key is caught by the third. +`@disabled-flags` is gated because it needs two things and only one of them comes for free. The +caller's default value is held by the provider, which always has it. What the provider also needs is +a **signal** that the flag was disabled, told apart from an ordinary resolution and from a missing +flag — and that belongs to the backend and its protocol. One with no disabled state, or one that +answers `FLAG_NOT_FOUND` for a disabled flag, gives the provider nothing to act on. + +[Appendix F][appendix-f] draws the line elsewhere — a provider whose backend decides, *"such as one +speaking OFREP, cannot: the server never sees the caller's default, so it has no way to return it"* — +and what this suite measured does not bear that out. flagd's RPC resolver is a remote evaluator by +exactly that description and satisfies the capability: the server answers reason `DISABLED` with no +variant and no value, and the resolver substitutes the caller's default locally on that signal. +flagd's OFREP endpoint answers the same flag with `{"reason": "DISABLED"}` and no `value` and no +`variant` — the same signal in another envelope — and the Python OFREP provider already falls back to +the caller's default for the absent value. It fails these scenarios for a reason unrelated to +architecture, which [its own suite](../../providers/openfeature-provider-ofrep/tests/tck/test_ofrep_conformance.py) +records. The discrepancy belongs upstream rather than papered over here; what it changes locally is +only what a withheld declaration may be read as — not necessarily an impossibility, so read the +adoption's own note for which it was. Withholding still needs no `KnownDeviation`, for the reason +every gated capability does: a deviation records a gap in behaviour the provider is *required* to +have, and this one is optional. + +Nothing in the specification says what a provider owes a disabled flag: +[Requirement 1.4.7](https://github.com/open-feature/spec/blob/main/specification/sections/01-flag-evaluation.md) +is about the SDK propagating whatever reason arrived, and 2.2.5 only lists `DISABLED` among the +reason strings a provider **may** use. So [Appendix F][appendix-f] states the behaviour, the way it +does for `@numeric-coercion`, and gates it. Since spec revision `009afe06` the canonical set carries +four `disabled-*` flags mirroring `boolean-flag`, `string-flag`, `integer-flag` and `float-flag` +exactly, differing only in `state`, and one Scenario Outline of four rows asserts that each resolves +to the caller's default. Each row's default differs from the flag's configured value, so a provider +that ignores the state is caught on the value alone — 2.2.3, a `MUST`. The rows assert neither the +reason, which would rest on 2.2.5's `SHOULD` and its "some other string", nor the variant, since a +disabled flag has resolved none: `@disabled-flags` and `@variants` deliberately do not compose. + Untagged scenarios are mandatory and always run. `capabilities` defaults to every *declarable* capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from the default, run the suite, and remove only what your provider genuinely cannot do. @@ -359,7 +393,7 @@ those scenarios are skipped with their reason. ## Findings -Three, all confirmed by running the suite rather than by reading code. +Four, all confirmed by running the suite rather than by reading code. ### 1. A boolean satisfies an Integer request @@ -397,6 +431,26 @@ This is not a defect: `@numeric-coercion` is optional, and the specification doe behaviour. So neither in-memory self-test declares the tag, and the three scenarios are skipped with that reason rather than failing. +### 4. The in-memory provider ignores a flag's state + +`InMemoryFlag` has a `State` enum with an `ENABLED` and a `DISABLED` member, takes one in its +constructor, and **never reads it**: `InMemoryFlag.resolve` returns the default variant's value with +reason `STATIC` whatever the state, and `InMemoryProvider._resolve` looks only for a missing key. So +all four `disabled-*` flags are served exactly as their enabled counterparts are. + +`canonical_flag_set` is not where this stops. `_decode_canonical_flags` reads the canonical file's +`"state": "DISABLED"`, validates it against `InMemoryFlag.State` and passes it through faithfully — +the self-tests pin that it reaches exactly those four flags and no others. The state survives +decoding and then has no effect. + +Measured before the tag was gated: all four rows failed on the value in both in-memory suites, with +`disabled-boolean-flag` resolving to `True` against a caller default of `false`. So neither suite +declares `@disabled-flags` and the four scenarios are skipped with that reason. + +Unlike finding 3 this is a field the SDK offers and does not honour, which is closer to a defect than +to a choice — but the capability is optional, so the honest report is still a withheld declaration +rather than a `KnownDeviation`. It is not filed against the SDK yet. + ## Where the assets come from The Gherkin feature files, the canonical flag set and the control-API document are **not owned by @@ -430,14 +484,14 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | --- | --- | --- | | `test_in_memory_conformance` | the SDK's `InMemoryProvider` | reference adoption for a backend-less provider | | `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | -| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set is decoded from `canonical-flags.json` — every flag served under the file's own default variant, with the Python type the file wrote | +| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set is decoded from `canonical-flags.json` — every flag served under the file's own default variant, with the Python type the file wrote, and `state` reaching exactly the four `disabled-*` flags | | `test_lifecycle_steps` | the steps that call the provider directly | the in-memory suites skip `@lifecycle`, so the shutdown, re-initialise and metadata steps are driven against a recording provider instead | | `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | | `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity | | `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping and the control-API it reports, against a stubbed control API | ``` -158 passed, 27 skipped, 2 xfailed +163 passed, 35 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; @@ -451,7 +505,9 @@ feature was gated on `@events`. Neither declares `@numeric-coercion` either, for finding 3, so its three scenarios are skipped too. Neither declares `@targeting`: both resolve the same decoded flag set, and `canonical_flag_set` deliberately ignores `targeting-key-flag`'s rule rather than becoming a second implementation of somebody else's evaluator, so those three scenarios -are skipped as well. Both declare `@variants`, since an in-memory flag set is keyed by variant name. +are skipped as well. Neither declares `@disabled-flags` either, for the reason in finding 4 — the +state reaches the flag set and the SDK's provider never reads it — so its four rows are skipped in +both. Both declare `@variants`, since an in-memory flag set is keyed by variant name. ## Known gaps diff --git a/tools/openfeature-provider-tck/spec b/tools/openfeature-provider-tck/spec index 26362f85b..009afe061 160000 --- a/tools/openfeature-provider-tck/spec +++ b/tools/openfeature-provider-tck/spec @@ -1 +1 @@ -Subproject commit 26362f85b7fcd59b35b969e6feebee80e206b24f +Subproject commit 009afe0617947121dcbebe4b66e0cc0c5cc4ada8 diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py index 13aaa3186..33c008a8d 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py +++ b/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py @@ -86,6 +86,75 @@ class Capability(str, Enum): makes the value a **MUST**. """ + DISABLED_FLAGS = "disabled-flags" + """Provider resolves a flag disabled in the management system to the code default. + + Gated because it needs two things and only one of them comes for free. The + caller's default value is held by the provider, which always has it. What + the provider also needs is a **signal** that the flag was disabled, told + apart from an ordinary resolution and from a missing flag -- and that is a + property of the backend and its protocol. One that has no disabled state, or + that answers ``FLAG_NOT_FOUND`` for a disabled flag, gives the provider + nothing to act on, and no care in the provider produces a substitution it + was never told to make. + + Appendix F draws the line somewhere else, and what this suite measured does + not bear that out. The appendix has it that a provider whose backend decides, + "such as one speaking OFREP, cannot: the server never sees the caller's + default, so it has no way to return it". Both halves of that are observably + not the obstacle. flagd's RPC resolver is a remote evaluator by exactly that + description and satisfies the capability: the server answers reason + ``DISABLED`` with no variant and no value, and the resolver substitutes the + caller's default locally on the strength of that signal + (``resolvers/grpc.py``). flagd's OFREP endpoint answers the same flag with + ``{"reason": "DISABLED"}`` and no ``value`` and no ``variant`` -- the same + signal in another envelope -- and the Python OFREP provider already falls + back to the caller's default for the absent value. It fails these scenarios + for a reason unrelated to architecture, which its own suite records. + + So the tag is worth gating, but for the reason above rather than the one the + appendix gives, and that discrepancy belongs upstream rather than papered + over here. What it changes locally is only what a withheld declaration may be + read as: not necessarily an impossibility, so a reader has to look at the + adoption's own note for which it was. + + Withholding it still needs no :class:`~.config.KnownDeviation`, for the + reason every gated capability does -- a deviation records a gap in behaviour + the provider is *required* to have, and this one is optional. That holds + whether the gap is architectural or a defect; where it is a defect, the + adoption's note is where to say so. + + Nothing in the specification says what a provider owes a disabled flag. + `Requirement 1.4.7 + `_ + is about the SDK propagating whatever reason arrived, and `Requirement 2.2.5 + `_ + only lists ``DISABLED`` among the reason strings a provider **may** use. So + Appendix F states the behaviour, the way it does for + :attr:`NUMERIC_COERCION`, and gates it. + + Declaring it runs one Scenario Outline of four rows, over the four + ``disabled-*`` flags the canonical set added at spec revision ``009afe06``. + They mirror ``boolean-flag``, ``string-flag``, ``integer-flag`` and + ``float-flag`` exactly, differing only in ``state``, and each row's caller + default differs from the flag's configured value -- so a provider that + ignores the state returns the configured value and is caught on the value + alone, which rests on 2.2.3, a **MUST**. + + The rows assert the value and the absence of an error, and deliberately + **not** the reason: pinning ``DISABLED`` would rest on 2.2.5, a **SHOULD** + that permits "some other string". No variant is asserted either, because a + disabled flag has resolved no variant and there is none to name -- so this + capability and :attr:`VARIANTS` do not compose, which is why the rows are not + part of the variant outline. + + The SDK's own ``InMemoryProvider`` cannot declare this, and the reason is + worth knowing before adopting it as a reference: ``InMemoryFlag`` accepts a + ``state`` of ``DISABLED`` and nothing ever reads it, so a disabled flag is + served like any other. ``_decode_canonical_flags`` passes the state through + faithfully; the provider is where it stops. + """ + UNAVAILABLE_INIT = "unavailable" """Provider reports an error state promptly against a backend it cannot reach.""" diff --git a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py index 5155ac803..8157a2e4d 100644 --- a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_controllable_conformance.py @@ -48,6 +48,12 @@ def tck_config() -> TckConfig: ``TARGETING`` stays undeclared for the reason given there as well: this is the same decoded flag set, and it ignores ``targeting-key-flag``'s rule. ``VARIANTS`` is declared, since the flag set is keyed by variant name. + + ``DISABLED_FLAGS`` stays undeclared for the reason given there too, and this + class inherits it rather than choosing it: ``ControllableInMemoryProvider`` + changes only how the flag set is *replaced*, and resolution -- including the + fact that ``InMemoryFlag.resolve`` never reads ``state`` -- is still the + SDK's. Measured the same way, with the same four failures. """ control = InProcessControl() return TckConfig( diff --git a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py index b5a6544e2..7f85674ce 100644 --- a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py @@ -102,6 +102,19 @@ def tck_config() -> TckConfig: vacuously while the feature was gated on ``EVENTS``, which is precisely the failure mode the split of ``@lifecycle`` from ``@events`` exists to end. A skip with a reason is the honest outcome. + * ``DISABLED_FLAGS`` -- omitted because the SDK's in-memory provider ignores + a flag's ``state``. ``InMemoryFlag`` has a ``State`` enum with a + ``DISABLED`` member, ``_decode_canonical_flags`` reads the canonical + file's ``"state": "DISABLED"`` and passes it through faithfully, and + ``InMemoryFlag.resolve`` never looks at it -- so all four ``disabled-*`` + flags are served at their own default variant with reason ``STATIC``, + where the scenarios expect the caller's default. Measured before it was + gated: the four rows failed on the value, ``disabled-boolean-flag`` + resolving to ``True`` against a caller default of ``false``. The + capability is optional, so the honest report is a withheld declaration + rather than a ``KnownDeviation`` -- but unlike ``NUMERIC_COERCION`` this + one is a field the SDK offers and does not honour, which is finding 4 in + the README. * ``NUMERIC_COERCION`` -- omitted because the SDK's in-memory provider does not coerce. It hands each variant back untouched, and the client's type check is ``isinstance``-based, so ``integral-float-flag`` (``10.0``) diff --git a/tools/openfeature-provider-tck/tests/test_in_process_control.py b/tools/openfeature-provider-tck/tests/test_in_process_control.py index 7545ee4af..2fe2d9434 100644 --- a/tools/openfeature-provider-tck/tests/test_in_process_control.py +++ b/tools/openfeature-provider-tck/tests/test_in_process_control.py @@ -27,6 +27,7 @@ from openfeature.contrib.tools.provider_tck.values import describe, values_equal from openfeature.event import ProviderEvent from openfeature.flag_evaluation import FlagType, Reason +from openfeature.provider.in_memory_provider import InMemoryFlag _NOT_SEEDED = "this default must never be what a seeded flag resolves to" """The default value handed to every resolver below. @@ -156,6 +157,35 @@ def _flag_type_of(value: typing.Any) -> FlagType: return FlagType.OBJECT +_DISABLED_FLAGS: tuple[tuple[str, str], ...] = ( + ("disabled-boolean-flag", "boolean-flag"), + ("disabled-string-flag", "string-flag"), + ("disabled-integer-flag", "integer-flag"), + ("disabled-float-flag", "float-flag"), +) +"""The flags the canonical file marks ``DISABLED``, paired with what each mirrors. + +Written out rather than derived from the file's ``state`` members, because the +tests below read those members to decide what to assert and a check that reads +the member deciding its own answer checks nothing. Four names and their +counterparts are cheap to keep; a silently empty set is not. +""" + +_IGNORES_STATE = ( + ". That flag is DISABLED in the canonical file and is served anyway, because " + "InMemoryFlag carries a state and InMemoryFlag.resolve never reads it -- " + "which is why neither in-memory suite declares @disabled-flags. If this row " + "has begun to fail, the SDK has started honouring DISABLED and the " + "capability has become declarable for both of them" +) +"""Appended to a failure about one of the four disabled flags. + +They are in the sweep below rather than excluded from it, so a reader who hits +one is told what the flag is and what its failing means, instead of finding an +exclusion list and no reason for it. +""" + + def test_every_packaged_flag_resolves_to_its_packaged_default_variant() -> None: """The whole of what seeding from the file has to achieve. @@ -168,9 +198,20 @@ def test_every_packaged_flag_resolves_to_its_packaged_default_variant() -> None: more than an equality check: a flag the seeding dropped resolves to the default value with no variant and ``FLAG_NOT_FOUND``, and for the falsy flags that fallback value can equal what was expected. + + **Including the four ``DISABLED`` flags, which is a finding rather than an + oversight.** Since spec revision ``009afe06`` the canonical set marks four + flags disabled, and the ``@disabled-flags`` scenarios expect each to resolve + to the caller's default. This provider resolves them to their own default + variant instead: the state survives decoding faithfully and + ``InMemoryFlag.resolve`` ignores it, so a disabled flag is indistinguishable + from an enabled one here. Sweeping them with everything else is therefore + the accurate statement of what this flag set does -- and the assertion that + obliges both in-memory suites to withhold the capability. """ canonical = json.loads(canonical_flags_json())["flags"] provider = ControllableInMemoryProvider(canonical_flag_set()) + ignores_state = {disabled for disabled, _ in _DISABLED_FLAGS} # Annotated explicitly, as in the evaluation step: the five typed resolvers # have different signatures, so an unannotated mapping infers a value type @@ -187,15 +228,16 @@ def test_every_packaged_flag_resolves_to_its_packaged_default_variant() -> None: for key, definition in canonical.items(): variant = definition["defaultVariant"] expected = definition["variants"][variant] + note = _IGNORES_STATE if key in ignores_state else "" details = resolvers[_flag_type_of(expected)](key, _NOT_SEEDED) - assert details.error_code is None, f"{key}: {details.error_message}" - assert details.variant == variant, key - assert details.reason == Reason.STATIC, key + assert details.error_code is None, f"{key}: {details.error_message}{note}" + assert details.variant == variant, f"{key}{note}" + assert details.reason == Reason.STATIC, f"{key}{note}" assert values_equal(expected, details.value), ( f"{key}/{variant}: packaged {describe(expected)}, " - f"resolved {describe(details.value)}" + f"resolved {describe(details.value)}{note}" ) @@ -277,6 +319,65 @@ def test_no_decoded_flag_carries_targeting() -> None: assert flag.context_evaluator is None, f"{key} has targeting" +def test_exactly_the_four_disabled_flags_are_decoded_disabled() -> None: + """``state`` reaches the flag set, and reaches only the four flags it should. + + Both halves are load-bearing. A decoder that dropped the member would leave + the canonical set with nothing disabled, and every scenario that assumes a + flag serves its own value would go on passing while ``@disabled-flags`` + became untestable; a state that leaked onto any other flag would break those + scenarios instead, which Appendix F calls out as the way to break the set + silently. + """ + decoded = canonical_flag_set() + disabled = { + key + for key, flag in decoded.items() + if flag.state is InMemoryFlag.State.DISABLED + } + + assert disabled == {key for key, _ in _DISABLED_FLAGS} + + +@pytest.mark.parametrize(("disabled", "enabled"), _DISABLED_FLAGS) +def test_a_disabled_flag_mirrors_its_enabled_counterpart( + disabled: str, enabled: str +) -> None: + """Differing only in state is what makes each row falsifiable. + + Every ``@disabled-flags`` row passes a caller default that is the flag's + *other* variant, so a provider ignoring the state returns the configured + value and fails on the value alone. Bring the two variants together -- give + ``disabled-string-flag`` a ``greeting`` of ``bye`` -- and the row passes + whether the state was honoured or not, which is the one way this outline can + be made vacuous without changing a single scenario. + + Stated as a mirror of the enabled counterpart rather than as four literal + values, because that is the property the file's own ``$comment`` and + Appendix F both claim, and it is the one a future edit would be reasoning + about. + """ + decoded = canonical_flag_set() + + assert decoded[disabled].variants == decoded[enabled].variants + assert decoded[disabled].default_variant == decoded[enabled].default_variant + assert decoded[enabled].state is InMemoryFlag.State.ENABLED + + variants = dict(decoded[disabled].variants) + served = variants.pop(decoded[disabled].default_variant) + assert len(variants) == 1, ( + f"{disabled} no longer offers exactly one other variant for a row's " + f"caller default to come from" + ) + [other] = variants.values() + # Compared the way the Then step compares, because that is the comparison a + # vacuous row would slip past. + assert not values_equal(served, other), ( + f"{disabled} would resolve {describe(served)} either way, so a provider " + f"that ignores DISABLED would pass its row" + ) + + def test_the_hand_built_changing_flag_matches_the_file() -> None: """``change_flag`` rebuilds ``changing-flag`` at its other variant. From 4803c085cfced66189323d2d1c878dc2b58b6782 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 13:46:59 +0200 Subject: [PATCH 23/46] refactor(tck)!: rename the package to openfeature-tck `provider-tck` named the package after what its contents test rather than after what the package is. The entry point is options-shaped, so a suite for something other than a provider can join this package later instead of a second `hook-tck` duplicating the harness -- which is erka's request on go-sdk-contrib#940, settled for all four languages at once so the naming stays in parity. directory tools/openfeature-provider-tck -> tools/openfeature-tck module openfeature.contrib.tools.provider_tck -> openfeature.contrib.tools.tck coordinate openfeature-provider-tck -> openfeature-tck Version stays 0.1.0: nothing is published anywhere, so this is free now and expensive later. A package rename here reaches outside the package. The workspace member list and `[tool.uv.sources]` in the root pyproject, the release-please config and manifest, the per-package filter in build.yml, the `.gitmodules` path and section name for the spec submodule, the two hatch build hooks that copy the spec assets in, the `pytest11` entry-point name and the gitignore for the copied assets all name the old path or distribution; uv.lock is regenerated. The OpenFeature domain the suite registers providers under follows, from `provider-tck/` to `tck/`: it is adopter-visible in test output and nothing asserts on it. The spec's own asset directory, `specification/assets/provider-tck/`, is not ours to rename and is deliberately left alone. Signed-off-by: Simon Schrottner --- .github/workflows/build.yml | 4 +- .gitmodules | 4 +- .release-please-manifest.json | 2 +- pyproject.toml | 6 +- release-please-config.json | 4 +- .../.gitignore | 6 +- .../LICENSE | 0 .../README.md | 18 +++-- .../hatch_build.py | 2 +- .../hatch_build_sync.py | 4 +- .../pyproject.toml | 10 +-- .../spec | 0 .../contrib/tools/tck}/__init__.py | 6 +- .../contrib/tools/tck}/capability.py | 0 .../openfeature/contrib/tools/tck}/config.py | 2 +- .../openfeature/contrib/tools/tck}/control.py | 0 .../contrib/tools/tck}/extensions.py | 2 +- .../contrib/tools/tck}/httpcontrol.py | 0 .../contrib/tools/tck}/inprocess.py | 0 .../openfeature/contrib/tools/tck}/plugin.py | 6 +- .../contrib/tools/tck}/provider.py | 2 +- .../openfeature/contrib/tools/tck}/state.py | 0 .../contrib/tools/tck}/steps/__init__.py | 0 .../contrib/tools/tck}/steps/event_steps.py | 0 .../contrib/tools/tck}/steps/flag_steps.py | 0 .../tools/tck}/steps/provider_steps.py | 0 .../openfeature/contrib/tools/tck}/values.py | 0 .../tests/conftest.py | 0 .../tests/test_controllable_conformance.py | 2 +- .../tests/test_declaration.py | 4 +- .../tests/test_extensions.py | 8 +-- .../tests/test_http_control.py | 2 +- .../tests/test_in_memory_conformance.py | 2 +- .../tests/test_in_process_control.py | 6 +- .../tests/test_lifecycle_steps.py | 6 +- uv.lock | 68 +++++++++---------- 36 files changed, 90 insertions(+), 86 deletions(-) rename tools/{openfeature-provider-tck => openfeature-tck}/.gitignore (62%) rename tools/{openfeature-provider-tck => openfeature-tck}/LICENSE (100%) rename tools/{openfeature-provider-tck => openfeature-tck}/README.md (98%) rename tools/{openfeature-provider-tck => openfeature-tck}/hatch_build.py (99%) rename tools/{openfeature-provider-tck => openfeature-tck}/hatch_build_sync.py (94%) rename tools/{openfeature-provider-tck => openfeature-tck}/pyproject.toml (89%) rename tools/{openfeature-provider-tck => openfeature-tck}/spec (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/__init__.py (95%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/capability.py (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/config.py (99%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/control.py (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/extensions.py (99%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/httpcontrol.py (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/inprocess.py (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/plugin.py (95%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/provider.py (99%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/state.py (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/steps/__init__.py (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/steps/event_steps.py (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/steps/flag_steps.py (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/steps/provider_steps.py (100%) rename tools/{openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck => openfeature-tck/src/openfeature/contrib/tools/tck}/values.py (100%) rename tools/{openfeature-provider-tck => openfeature-tck}/tests/conftest.py (100%) rename tools/{openfeature-provider-tck => openfeature-tck}/tests/test_controllable_conformance.py (98%) rename tools/{openfeature-provider-tck => openfeature-tck}/tests/test_declaration.py (99%) rename tools/{openfeature-provider-tck => openfeature-tck}/tests/test_extensions.py (98%) rename tools/{openfeature-provider-tck => openfeature-tck}/tests/test_http_control.py (99%) rename tools/{openfeature-provider-tck => openfeature-tck}/tests/test_in_memory_conformance.py (99%) rename tools/{openfeature-provider-tck => openfeature-tck}/tests/test_in_process_control.py (99%) rename tools/{openfeature-provider-tck => openfeature-tck}/tests/test_lifecycle_steps.py (98%) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 59b8f0df3..d59c38f7d 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -73,8 +73,8 @@ jobs: tools/openfeature-flagd-api-testkit: - 'tools/openfeature-flagd-api-testkit/**' - 'uv.lock' - tools/openfeature-provider-tck: - - 'tools/openfeature-provider-tck/**' + tools/openfeature-tck: + - 'tools/openfeature-tck/**' - 'uv.lock' build: diff --git a/.gitmodules b/.gitmodules index 31678c42e..637a63b16 100644 --- a/.gitmodules +++ b/.gitmodules @@ -7,6 +7,6 @@ [submodule "providers/openfeature-provider-flagd/openfeature/test-harness"] path = providers/openfeature-provider-flagd/openfeature/test-harness url = https://github.com/open-feature/flagd-testbed.git -[submodule "tools/openfeature-provider-tck/spec"] - path = tools/openfeature-provider-tck/spec +[submodule "tools/openfeature-tck/spec"] + path = tools/openfeature-tck/spec url = https://github.com/open-feature/spec diff --git a/.release-please-manifest.json b/.release-please-manifest.json index 6bb086204..2c0bdfd17 100644 --- a/.release-please-manifest.json +++ b/.release-please-manifest.json @@ -9,5 +9,5 @@ "tools/openfeature-flagd-api": "1.0.0", "tools/openfeature-flagd-core": "1.0.0", "tools/openfeature-flagd-api-testkit": "0.1.0", - "tools/openfeature-provider-tck": "0.1.0" + "tools/openfeature-tck": "0.1.0" } diff --git a/pyproject.toml b/pyproject.toml index e250a4b79..244df6310 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -16,7 +16,7 @@ dependencies = [ "openfeature-flagd-api", "openfeature-flagd-core", "openfeature-flagd-api-testkit", - "openfeature-provider-tck", + "openfeature-tck", ] [dependency-groups] @@ -44,7 +44,7 @@ openfeature-provider-unleash = { workspace = true } openfeature-flagd-api = { workspace = true } openfeature-flagd-core = { workspace = true } openfeature-flagd-api-testkit = { workspace = true } -openfeature-provider-tck = { workspace = true } +openfeature-tck = { workspace = true } [tool.uv.workspace] members = [ @@ -63,7 +63,7 @@ exclude = [ "providers/openfeature-provider-flagd/src/openfeature/schemas/**", # Submodules of other repositories: not ours to lint or format. "providers/openfeature-provider-flagd/openfeature/spec/**", - "tools/openfeature-provider-tck/spec/**", + "tools/openfeature-tck/spec/**", ] [tool.ruff.lint] diff --git a/release-please-config.json b/release-please-config.json index 29cfce44b..f890331ff 100644 --- a/release-please-config.json +++ b/release-please-config.json @@ -100,8 +100,8 @@ "README.md" ] }, - "tools/openfeature-provider-tck": { - "package-name": "openfeature-provider-tck", + "tools/openfeature-tck": { + "package-name": "openfeature-tck", "bump-minor-pre-major": true, "bump-patch-for-minor-pre-major": true, "versioning": "default", diff --git a/tools/openfeature-provider-tck/.gitignore b/tools/openfeature-tck/.gitignore similarity index 62% rename from tools/openfeature-provider-tck/.gitignore rename to tools/openfeature-tck/.gitignore index 72a17508a..41520f2e8 100644 --- a/tools/openfeature-provider-tck/.gitignore +++ b/tools/openfeature-tck/.gitignore @@ -2,6 +2,6 @@ # DO NOT EDIT the copies, and do not commit them: the canonical definitions live # in spec/specification/assets/provider-tck/, and the revision this package is # built against is recorded by the submodule pin. -src/openfeature/contrib/tools/provider_tck/gherkin/ -src/openfeature/contrib/tools/provider_tck/flag_data/ -src/openfeature/contrib/tools/provider_tck/control-api.yaml +src/openfeature/contrib/tools/tck/gherkin/ +src/openfeature/contrib/tools/tck/flag_data/ +src/openfeature/contrib/tools/tck/control-api.yaml diff --git a/tools/openfeature-provider-tck/LICENSE b/tools/openfeature-tck/LICENSE similarity index 100% rename from tools/openfeature-provider-tck/LICENSE rename to tools/openfeature-tck/LICENSE diff --git a/tools/openfeature-provider-tck/README.md b/tools/openfeature-tck/README.md similarity index 98% rename from tools/openfeature-provider-tck/README.md rename to tools/openfeature-tck/README.md index 6e8ff5337..2dfc989dc 100644 --- a/tools/openfeature-provider-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -1,8 +1,12 @@ -# OpenFeature Provider TCK (Python) +# OpenFeature TCK (Python) A conformance suite any OpenFeature Python provider can adopt to verify that it implements the provider contract of the specification. +Named `tck` rather than `provider-tck` because the name should say what the package *is*, not what +its current contents test: the entry point is options-shaped, so a suite for something other than a +provider can join it later instead of a second package duplicating the harness. + OpenFeature's central promise is that swapping providers does not change application behaviour. Nothing verifies that today, and every provider tests differently — so "implements the provider contract" is an unverified claim, and a behavioural difference between two providers is discovered @@ -29,7 +33,7 @@ testkit already use, so an adopting package gains no new test framework. import pytest from pytest_bdd import scenarios -from openfeature.contrib.tools.provider_tck import ( +from openfeature.contrib.tools.tck import ( Capability, TckConfig, feature_paths, @@ -92,7 +96,7 @@ tests/ # conftest.py from pytest_bdd import then -from openfeature.contrib.tools.provider_tck import TckState +from openfeature.contrib.tools.tck import TckState @then("the fractional rule splits the population") @@ -460,14 +464,14 @@ the same ones — which is the only reason a conformance claim means the same th does in Java. **Adopting this package needs no submodule.** The assets are copied into the wheel and the sdist at -build time, so `pip install openfeature-provider-tck` gives you everything the suite runs on. +build time, so `pip install openfeature-tck` gives you everything the suite runs on. **Contributing to this package does.** The spec is a git submodule at -`tools/openfeature-provider-tck/spec`, and the copies under -`src/openfeature/contrib/tools/provider_tck/` are gitignored and generated: +`tools/openfeature-tck/spec`, and the copies under +`src/openfeature/contrib/tools/tck/` are gitignored and generated: ```bash -git submodule update --init tools/openfeature-provider-tck/spec +git submodule update --init tools/openfeature-tck/spec poe test # runs `poe sync-spec-assets` first ``` diff --git a/tools/openfeature-provider-tck/hatch_build.py b/tools/openfeature-tck/hatch_build.py similarity index 99% rename from tools/openfeature-provider-tck/hatch_build.py rename to tools/openfeature-tck/hatch_build.py index 4b6f1d840..5bff5a40e 100644 --- a/tools/openfeature-provider-tck/hatch_build.py +++ b/tools/openfeature-tck/hatch_build.py @@ -38,7 +38,7 @@ def initialize(self, version: str, build_data: dict) -> None: msg = ( f"Conformance assets missing ({missing}) and the open-feature/spec " f"submodule is not checked out at {SPEC_ASSETS}. Run " - "`git submodule update --init tools/openfeature-provider-tck/spec`." + "`git submodule update --init tools/openfeature-tck/spec`." ) raise FileNotFoundError(msg) diff --git a/tools/openfeature-provider-tck/hatch_build_sync.py b/tools/openfeature-tck/hatch_build_sync.py similarity index 94% rename from tools/openfeature-provider-tck/hatch_build_sync.py rename to tools/openfeature-tck/hatch_build_sync.py index 102590882..054f5ca65 100644 --- a/tools/openfeature-provider-tck/hatch_build_sync.py +++ b/tools/openfeature-tck/hatch_build_sync.py @@ -15,7 +15,7 @@ ROOT = Path(__file__).parent SPEC_ASSETS = (ROOT / "spec/specification/assets/provider-tck").resolve() -PACKAGE_REL = Path("src/openfeature/contrib/tools/provider_tck") +PACKAGE_REL = Path("src/openfeature/contrib/tools/tck") DEST_BASE = ROOT / PACKAGE_REL DO_NOT_EDIT = ( @@ -41,7 +41,7 @@ def sync() -> None: msg = ( f"Conformance assets not found at {SPEC_ASSETS}. " "Make sure submodules are initialized: " - "`git submodule update --init tools/openfeature-provider-tck/spec`." + "`git submodule update --init tools/openfeature-tck/spec`." ) raise FileNotFoundError(msg) diff --git a/tools/openfeature-provider-tck/pyproject.toml b/tools/openfeature-tck/pyproject.toml similarity index 89% rename from tools/openfeature-provider-tck/pyproject.toml rename to tools/openfeature-tck/pyproject.toml index 3825d97b5..28886de94 100644 --- a/tools/openfeature-provider-tck/pyproject.toml +++ b/tools/openfeature-tck/pyproject.toml @@ -3,7 +3,7 @@ requires = ["hatchling"] build-backend = "hatchling.build" [project] -name = "openfeature-provider-tck" +name = "openfeature-tck" version = "0.1.0" description = "OpenFeature provider conformance suite (TCK)" readme = "README.md" @@ -33,7 +33,7 @@ Homepage = "https://github.com/open-feature/python-sdk-contrib" # fixtures from an installed plugin are visible to every test, so an adopter # never has to `from ... import *` to pull the vocabulary in. [project.entry-points.pytest11] -openfeature_provider_tck = "openfeature.contrib.tools.provider_tck.plugin" +openfeature_tck = "openfeature.contrib.tools.tck.plugin" [dependency-groups] dev = [ @@ -55,9 +55,9 @@ packages = ["src/openfeature"] # Ship the conformance assets even though they are gitignored: an adopter # installing this package must need no submodule of their own. artifacts = [ - "src/openfeature/contrib/tools/provider_tck/gherkin/", - "src/openfeature/contrib/tools/provider_tck/flag_data/", - "src/openfeature/contrib/tools/provider_tck/control-api.yaml", + "src/openfeature/contrib/tools/tck/gherkin/", + "src/openfeature/contrib/tools/tck/flag_data/", + "src/openfeature/contrib/tools/tck/control-api.yaml", ] [tool.hatch.build.hooks.custom] diff --git a/tools/openfeature-provider-tck/spec b/tools/openfeature-tck/spec similarity index 100% rename from tools/openfeature-provider-tck/spec rename to tools/openfeature-tck/spec diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py similarity index 95% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py index 18cc4f341..24d279e59 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/__init__.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py @@ -12,7 +12,7 @@ import pytest from pytest_bdd import scenarios - from openfeature.contrib.tools.provider_tck import ( + from openfeature.contrib.tools.tck import ( Capability, InProcessControl, TckConfig, @@ -108,7 +108,7 @@ def tck_config(): # by this repository and are NOT committed to it. They are copies of the # language-agnostic conformance artifacts defined in open-feature/spec under # specification/assets/provider-tck/, which reaches this package as a git -# submodule at tools/openfeature-provider-tck/spec and is copied in at build +# submodule at tools/openfeature-tck/spec and is copied in at build # time by hatch_build.py. The copies are gitignored, so the only record of which # spec revision this package targets is the submodule pin, and the two cannot # drift apart unnoticed. @@ -121,7 +121,7 @@ def tck_config(): # the one thing this suite exists to prevent. # See https://github.com/open-feature/spec/issues/417. -_PACKAGE = "openfeature.contrib.tools.provider_tck" +_PACKAGE = "openfeature.contrib.tools.tck" def control_api_spec() -> str: diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/capability.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py similarity index 99% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py index ed0775b7e..6ff454584 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/config.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py @@ -232,7 +232,7 @@ def domain(self) -> str: for a provider holding a network connection means leaking one connection per scenario. """ - return f"provider-tck/{self.name}" + return f"tck/{self.name}" def declares(self, capability: Capability) -> bool: return capability in self.capabilities diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/control.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/control.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/control.py diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py similarity index 99% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py index 26849b811..ea19b1b1a 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/extensions.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py @@ -68,7 +68,7 @@ "uri_for", ] -_PACKAGE = "openfeature.contrib.tools.provider_tck" +_PACKAGE = "openfeature.contrib.tools.tck" CANONICAL_DIRECTORY = "gherkin" """The packaged directory the canonical feature files live in. diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/httpcontrol.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/httpcontrol.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/inprocess.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/inprocess.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/inprocess.py diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py similarity index 95% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py index d4b66bb4b..18103e981 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/plugin.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py @@ -30,9 +30,9 @@ # module defining it is a registered plugin -- importing it here would run the # decorators but leave those fixtures where pytest never looks. pytest_plugins = [ - "openfeature.contrib.tools.provider_tck.steps.provider_steps", - "openfeature.contrib.tools.provider_tck.steps.flag_steps", - "openfeature.contrib.tools.provider_tck.steps.event_steps", + "openfeature.contrib.tools.tck.steps.provider_steps", + "openfeature.contrib.tools.tck.steps.flag_steps", + "openfeature.contrib.tools.tck.steps.event_steps", ] diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/provider.py similarity index 99% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/provider.py index a0c273aa4..636b7e66c 100644 --- a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/provider.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/provider.py @@ -27,7 +27,7 @@ _CHANGING_BASELINE = "foo" _CHANGING_CHANGED = "bar" -_PACKAGE = "openfeature.contrib.tools.provider_tck" +_PACKAGE = "openfeature.contrib.tools.tck" _FLAG_DATA_DIRECTORY = "flag_data" _CANONICAL_FLAGS_FILE = "canonical-flags.json" diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/state.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/state.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/state.py diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/__init__.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/steps/__init__.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/__init__.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/steps/__init__.py diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/steps/event_steps.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/event_steps.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/steps/event_steps.py diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/steps/flag_steps.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/flag_steps.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/steps/flag_steps.py diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/steps/provider_steps.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/steps/provider_steps.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/steps/provider_steps.py diff --git a/tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/values.py similarity index 100% rename from tools/openfeature-provider-tck/src/openfeature/contrib/tools/provider_tck/values.py rename to tools/openfeature-tck/src/openfeature/contrib/tools/tck/values.py diff --git a/tools/openfeature-provider-tck/tests/conftest.py b/tools/openfeature-tck/tests/conftest.py similarity index 100% rename from tools/openfeature-provider-tck/tests/conftest.py rename to tools/openfeature-tck/tests/conftest.py diff --git a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py b/tools/openfeature-tck/tests/test_controllable_conformance.py similarity index 98% rename from tools/openfeature-provider-tck/tests/test_controllable_conformance.py rename to tools/openfeature-tck/tests/test_controllable_conformance.py index 8157a2e4d..0bad9c54c 100644 --- a/tools/openfeature-provider-tck/tests/test_controllable_conformance.py +++ b/tools/openfeature-tck/tests/test_controllable_conformance.py @@ -17,7 +17,7 @@ import pytest from pytest_bdd import scenarios -from openfeature.contrib.tools.provider_tck import ( +from openfeature.contrib.tools.tck import ( Capability, InProcessControl, TckConfig, diff --git a/tools/openfeature-provider-tck/tests/test_declaration.py b/tools/openfeature-tck/tests/test_declaration.py similarity index 99% rename from tools/openfeature-provider-tck/tests/test_declaration.py rename to tools/openfeature-tck/tests/test_declaration.py index 9c4b19200..8af5eb9a5 100644 --- a/tools/openfeature-provider-tck/tests/test_declaration.py +++ b/tools/openfeature-tck/tests/test_declaration.py @@ -27,7 +27,7 @@ import pytest -from openfeature.contrib.tools.provider_tck import ( +from openfeature.contrib.tools.tck import ( DECLARABLE_CAPABILITIES, RESERVED_CAPABILITIES, BackendControl, @@ -37,7 +37,7 @@ TckConfig, features_path, ) -from openfeature.contrib.tools.provider_tck.capability import ( +from openfeature.contrib.tools.tck.capability import ( capability_for_marker, capability_for_tag, ) diff --git a/tools/openfeature-provider-tck/tests/test_extensions.py b/tools/openfeature-tck/tests/test_extensions.py similarity index 98% rename from tools/openfeature-provider-tck/tests/test_extensions.py rename to tools/openfeature-tck/tests/test_extensions.py index a6e95d65e..ab4883d85 100644 --- a/tools/openfeature-provider-tck/tests/test_extensions.py +++ b/tools/openfeature-tck/tests/test_extensions.py @@ -39,12 +39,12 @@ import pytest -from openfeature.contrib.tools.provider_tck import ( +from openfeature.contrib.tools.tck import ( EXTENSIONS_DIRECTORY, feature_paths, features_path, ) -from openfeature.contrib.tools.provider_tck.extensions import ( +from openfeature.contrib.tools.tck.extensions import ( CANONICAL_DIRECTORY, EXTENSIONS_URI_PREFIX, collision_problem, @@ -71,7 +71,7 @@ import pytest from pytest_bdd import scenarios -from openfeature.contrib.tools.provider_tck import ( +from openfeature.contrib.tools.tck import ( Capability, InProcessControl, TckConfig, @@ -106,7 +106,7 @@ def tck_config(): import pytest from pytest_bdd import then -from openfeature.contrib.tools.provider_tck import TckState +from openfeature.contrib.tools.tck import TckState DEVIATION = "[boolean-flag-Integer-1]" diff --git a/tools/openfeature-provider-tck/tests/test_http_control.py b/tools/openfeature-tck/tests/test_http_control.py similarity index 99% rename from tools/openfeature-provider-tck/tests/test_http_control.py rename to tools/openfeature-tck/tests/test_http_control.py index e80523864..09b9d03f1 100644 --- a/tools/openfeature-provider-tck/tests/test_http_control.py +++ b/tools/openfeature-tck/tests/test_http_control.py @@ -19,7 +19,7 @@ import pytest -from openfeature.contrib.tools.provider_tck import ( +from openfeature.contrib.tools.tck import ( ControlApiError, HttpControl, ) diff --git a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py b/tools/openfeature-tck/tests/test_in_memory_conformance.py similarity index 99% rename from tools/openfeature-provider-tck/tests/test_in_memory_conformance.py rename to tools/openfeature-tck/tests/test_in_memory_conformance.py index 7f85674ce..b4a256e3d 100644 --- a/tools/openfeature-provider-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-tck/tests/test_in_memory_conformance.py @@ -20,7 +20,7 @@ import pytest from pytest_bdd import scenarios -from openfeature.contrib.tools.provider_tck import ( +from openfeature.contrib.tools.tck import ( Capability, TckConfig, canonical_flag_set, diff --git a/tools/openfeature-provider-tck/tests/test_in_process_control.py b/tools/openfeature-tck/tests/test_in_process_control.py similarity index 99% rename from tools/openfeature-provider-tck/tests/test_in_process_control.py rename to tools/openfeature-tck/tests/test_in_process_control.py index 2fe2d9434..df890e14c 100644 --- a/tools/openfeature-provider-tck/tests/test_in_process_control.py +++ b/tools/openfeature-tck/tests/test_in_process_control.py @@ -12,7 +12,7 @@ import pytest -from openfeature.contrib.tools.provider_tck import ( +from openfeature.contrib.tools.tck import ( CHANGING_FLAG_KEY, ConnectionControl, ControllableInMemoryProvider, @@ -20,11 +20,11 @@ canonical_flag_set, canonical_flags_json, ) -from openfeature.contrib.tools.provider_tck.provider import ( +from openfeature.contrib.tools.tck.provider import ( _decode_canonical_flags, changing_flag, ) -from openfeature.contrib.tools.provider_tck.values import describe, values_equal +from openfeature.contrib.tools.tck.values import describe, values_equal from openfeature.event import ProviderEvent from openfeature.flag_evaluation import FlagType, Reason from openfeature.provider.in_memory_provider import InMemoryFlag diff --git a/tools/openfeature-provider-tck/tests/test_lifecycle_steps.py b/tools/openfeature-tck/tests/test_lifecycle_steps.py similarity index 98% rename from tools/openfeature-provider-tck/tests/test_lifecycle_steps.py rename to tools/openfeature-tck/tests/test_lifecycle_steps.py index 9f2024395..350b38d02 100644 --- a/tools/openfeature-provider-tck/tests/test_lifecycle_steps.py +++ b/tools/openfeature-tck/tests/test_lifecycle_steps.py @@ -24,19 +24,19 @@ import pytest from openfeature import api -from openfeature.contrib.tools.provider_tck import ( +from openfeature.contrib.tools.tck import ( Capability, TckConfig, TckState, canonical_flag_set, ) -from openfeature.contrib.tools.provider_tck.steps.flag_steps import ( +from openfeature.contrib.tools.tck.steps.flag_steps import ( a_flag_with_key_and_default, no_exception_should_have_been_thrown, the_flag_was_evaluated_with_details, the_resolved_value_should_be, ) -from openfeature.contrib.tools.provider_tck.steps.provider_steps import ( +from openfeature.contrib.tools.tck.steps.provider_steps import ( a_stable_provider, the_provider_is_initialized_again, the_provider_is_shut_down, diff --git a/uv.lock b/uv.lock index 804640a93..4c723c6fe 100644 --- a/uv.lock +++ b/uv.lock @@ -17,9 +17,9 @@ members = [ "openfeature-provider-flagd", "openfeature-provider-flipt", "openfeature-provider-ofrep", - "openfeature-provider-tck", "openfeature-provider-unleash", "openfeature-python-contrib", + "openfeature-tck", ] [[package]] @@ -1990,37 +1990,6 @@ dev = [ { name = "types-requests", specifier = ">=2.32.0,<3.0.0" }, ] -[[package]] -name = "openfeature-provider-tck" -version = "0.1.0" -source = { editable = "tools/openfeature-provider-tck" } -dependencies = [ - { name = "openfeature-sdk" }, - { name = "pytest" }, - { name = "pytest-bdd" }, -] - -[package.dev-dependencies] -dev = [ - { name = "coverage", extra = ["toml"] }, - { name = "mypy" }, - { name = "poethepoet" }, -] - -[package.metadata] -requires-dist = [ - { name = "openfeature-sdk", specifier = ">=0.10.0" }, - { name = "pytest", specifier = ">=8.4.0" }, - { name = "pytest-bdd", specifier = ">=8.1.0,<9.0.0" }, -] - -[package.metadata.requires-dev] -dev = [ - { name = "coverage", extras = ["toml"], specifier = ">=7.10.0,<8.0.0" }, - { name = "mypy", specifier = ">=1.18.0,<2.0.0" }, - { name = "poethepoet", specifier = ">=0.37.0" }, -] - [[package]] name = "openfeature-provider-unleash" version = "0.1.2" @@ -2074,8 +2043,8 @@ dependencies = [ { name = "openfeature-provider-flagd" }, { name = "openfeature-provider-flipt" }, { name = "openfeature-provider-ofrep" }, - { name = "openfeature-provider-tck" }, { name = "openfeature-provider-unleash" }, + { name = "openfeature-tck" }, ] [package.dev-dependencies] @@ -2096,8 +2065,8 @@ requires-dist = [ { name = "openfeature-provider-flagd", editable = "providers/openfeature-provider-flagd" }, { name = "openfeature-provider-flipt", editable = "providers/openfeature-provider-flipt" }, { name = "openfeature-provider-ofrep", editable = "providers/openfeature-provider-ofrep" }, - { name = "openfeature-provider-tck", editable = "tools/openfeature-provider-tck" }, { name = "openfeature-provider-unleash", editable = "providers/openfeature-provider-unleash" }, + { name = "openfeature-tck", editable = "tools/openfeature-tck" }, ] [package.metadata.requires-dev] @@ -2116,6 +2085,37 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/da/44/8a4f5225e930ff0d999fd43f5d743a4babeb6c7e76dddc00f0e118878ef3/openfeature_sdk-0.10.0-py3-none-any.whl", hash = "sha256:75497ea75d73f684eef509a25f79ad6386368862e050af80ab70a44ae49b33e4", size = 38941, upload-time = "2026-06-01T19:45:33.011Z" }, ] +[[package]] +name = "openfeature-tck" +version = "0.1.0" +source = { editable = "tools/openfeature-tck" } +dependencies = [ + { name = "openfeature-sdk" }, + { name = "pytest" }, + { name = "pytest-bdd" }, +] + +[package.dev-dependencies] +dev = [ + { name = "coverage", extra = ["toml"] }, + { name = "mypy" }, + { name = "poethepoet" }, +] + +[package.metadata] +requires-dist = [ + { name = "openfeature-sdk", specifier = ">=0.10.0" }, + { name = "pytest", specifier = ">=8.4.0" }, + { name = "pytest-bdd", specifier = ">=8.1.0,<9.0.0" }, +] + +[package.metadata.requires-dev] +dev = [ + { name = "coverage", extras = ["toml"], specifier = ">=7.10.0,<8.0.0" }, + { name = "mypy", specifier = ">=1.18.0,<2.0.0" }, + { name = "poethepoet", specifier = ">=0.37.0" }, +] + [[package]] name = "opentelemetry-api" version = "1.41.1" From c5bd907946792849e0a54b313c7c6f20eaaadee2 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 14:05:50 +0200 Subject: [PATCH 24/46] refactor(tck)!: remove features_path(), which silently dropped extensions `features_path()` and `feature_paths()` were both public. They differ by one character at the call site, both are valid arguments to `scenarios()`, and the first one returns the canonical set alone -- so an adopter who reached for it got a green run over fewer scenarios than they believed had run. That is the worst failure mode available to a conformance suite, because unlike a red run there is nothing there to notice it. It was not hypothetical: both flagd resolver suites and the OFREP suite were calling it, so three of the three real adoptions were quietly unable to contribute an extension scenario. Removed outright rather than deprecated. Nothing is published, so there is no deprecation obligation, and a deprecated name that still works is still a name someone writes. `canonical_root()` is the supported way to reach the packaged directory for anything that is not "the scenarios to run"; the path-returning half survives as a private `_canonical_path()`. The baseline half of the extension self-test used to be the `features_path()` call. It is now a module one directory below the `extensions` directory, which is a better test of the same property: an extension belongs to the module it sits beside, and the directory a module is in -- not the session it runs in -- is what decides what it collects. Signed-off-by: Simon Schrottner --- .../contrib/tools/tck/extensions.py | 23 ++++-- .../tests/test_controllable_conformance.py | 4 +- .../openfeature-tck/tests/test_extensions.py | 79 +++++++++++++------ .../tests/test_in_memory_conformance.py | 4 +- 4 files changed, 74 insertions(+), 36 deletions(-) diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py index ea19b1b1a..83617f2b9 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py @@ -60,7 +60,6 @@ "collision_problem", "extension_root", "feature_paths", - "features_path", "is_canonical", "is_canonical_uri", "reserved_prefix_problem", @@ -115,12 +114,20 @@ """ -def features_path() -> str: - """Return the directory holding the canonical feature files. +def _canonical_path() -> str: + """The packaged directory holding the canonical feature files. - Packaged with this distribution, so a consumer needs no submodule and no - particular directory layout. This is the canonical set on its own; prefer - :func:`feature_paths`, which also picks up an adopter's own scenarios. + Deliberately not public. It used to be, as ``features_path()``, and the + public pair was a trap: ``scenarios(features_path())`` and + ``scenarios(*feature_paths())`` are both valid calls, differ by one + character at the call site, and the first one silently drops the extensions + directory. What that produces is a green run that examined fewer scenarios + than the adopter believes it did, which is the worst failure mode available + to a conformance suite -- worse than a red one, because nothing is there to + notice. Both flagd suites and the OFREP suite were calling it. + + :func:`canonical_root` is the supported way to reach the directory for + anything that is not "the scenarios to run". """ return str(importlib.resources.files(_PACKAGE) / CANONICAL_DIRECTORY) @@ -145,7 +152,7 @@ def feature_paths() -> tuple[str, ...]: with no ``__file__`` -- an interactive session, an exec'd string -- gets the canonical set alone. """ - paths = [features_path()] + paths = [_canonical_path()] directory = _caller_directory() if directory is not None: extensions = extension_root(directory) @@ -173,7 +180,7 @@ def canonical_root() -> Path | None: the honest answer and never a false accusation. """ try: - return _resolve(Path(features_path())) + return _resolve(Path(_canonical_path())) except (OSError, TypeError): # pragma: no cover - assets outside a filesystem return None diff --git a/tools/openfeature-tck/tests/test_controllable_conformance.py b/tools/openfeature-tck/tests/test_controllable_conformance.py index 0bad9c54c..06fa78725 100644 --- a/tools/openfeature-tck/tests/test_controllable_conformance.py +++ b/tools/openfeature-tck/tests/test_controllable_conformance.py @@ -21,7 +21,7 @@ Capability, InProcessControl, TckConfig, - features_path, + feature_paths, ) @@ -70,4 +70,4 @@ class inherits it rather than choosing it: ``ControllableInMemoryProvider`` ) -scenarios(features_path()) +scenarios(*feature_paths()) diff --git a/tools/openfeature-tck/tests/test_extensions.py b/tools/openfeature-tck/tests/test_extensions.py index ab4883d85..a84bbeb0d 100644 --- a/tools/openfeature-tck/tests/test_extensions.py +++ b/tools/openfeature-tck/tests/test_extensions.py @@ -41,8 +41,8 @@ from openfeature.contrib.tools.tck import ( EXTENSIONS_DIRECTORY, + canonical_root, feature_paths, - features_path, ) from openfeature.contrib.tools.tck.extensions import ( CANONICAL_DIRECTORY, @@ -56,6 +56,24 @@ uri_for, ) + +def _canonical_root() -> Path: + """The packaged canonical directory, or a failure that says how to get one. + + ``canonical_root()`` answers ``None`` when the assets are not on a + filesystem, which is the honest answer for a zipimport and a missing build + step everywhere else. + """ + root = canonical_root() + assert root is not None, ( + "the packaged canonical features must be on a filesystem for this file " + "to have anything to say; run `poe sync-spec-assets` first" + ) + return root + + +CANONICAL_ROOT = _canonical_root() + CANONICAL_FEATURE = "errors.feature" """The canonical file the collision cases are written against, chosen because it is the one whose scenarios an extension could most plausibly want to restate.""" @@ -76,7 +94,6 @@ InProcessControl, TckConfig, feature_paths, - features_path, ) @@ -91,12 +108,9 @@ def tck_config(): ) -scenarios({call}) +scenarios(*feature_paths()) ''' -_EXTENSION_CALL = "*feature_paths()" -_CANONICAL_CALL = "features_path()" - # The step the adopter writes, in the adopter's own conftest.py and nowhere else. # It asks the TCK's own per-scenario state what happened, which is what makes # "the same session and the same provider" checkable rather than asserted: a @@ -217,20 +231,35 @@ def _run( return Run(directory=directory, result=result, outcomes=outcomes) -def _suite(name: str, call: str = _EXTENSION_CALL) -> str: - return _SUITE_MODULE.format(name=name, call=call) +def _suite(name: str) -> str: + return _SUITE_MODULE.format(name=name) + + +BASELINE_DIRECTORY = "baseline" +"""Where the adoption that has no extensions of its own lives. + +A subdirectory rather than a second module beside the first, because both +adoptions now write the *same* line -- ``scenarios(*feature_paths())`` -- and +what distinguishes them is where they are written. ``feature_paths()`` looks for +an ``extensions`` directory beside the calling module, so a module one level +down is an adopter with none, in a session where one exists a directory away. + +There used to be a second public call, ``features_path()``, which returned the +canonical set alone and was what ``before`` used. It is gone: the two calls +differed by one character and the shorter one silently dropped the extensions +directory, so an adopter who reached for it got a green run over fewer scenarios +than they believed they had run. +""" @pytest.fixture(scope="module") def adoption(tmp_path_factory: pytest.TempPathFactory) -> Run: """One session running two adoptions of the same provider. - ``before`` is the call an adopter wrote before any of this, - ``scenarios(features_path())``, which sees no extension however many are - lying beside it. ``after`` is ``scenarios(*feature_paths())`` with an - ordinary extension beside it. Having both in one run is what lets "an - extension adds and does not alter" be a comparison rather than a number - written down here. + ``before`` is an adopter with no extensions directory of their own; ``after`` + is the same adoption with an ordinary extension beside it. Having both in one + run is what lets "an extension adds and does not alter" be a comparison + rather than a number written down here. One session rather than two, because a subprocess pytest run is by far the most expensive thing in this file and the two suites are independent: each @@ -239,8 +268,8 @@ def adoption(tmp_path_factory: pytest.TempPathFactory) -> Run: return _run( tmp_path_factory, { - "test_before.py": _suite("before", _CANONICAL_CALL), - "test_after.py": _suite("after", _EXTENSION_CALL), + f"{BASELINE_DIRECTORY}/test_before.py": _suite("before"), + "test_after.py": _suite("after"), "conftest.py": _CONFTEST_MODULE, }, {f"{EXTENSIONS_DIRECTORY}/vendor.feature": _VENDOR_FEATURE}, @@ -336,16 +365,18 @@ def test_an_extension_adds_scenarios_and_alters_none(adoption: Run) -> None: assert {name: after[name] for name in before} == before -def test_features_path_sees_no_extension_however_many_are_beside_it( +def test_a_module_with_no_extensions_directory_beside_it_sees_no_extension( adoption: Run, ) -> None: - """The older call still means exactly what it meant: the canonical set. + """An extension belongs to the module it sits beside, and to no other. - Both generated suites sit in the same directory as the ``extensions`` - directory, so the one that asks for ``features_path()`` is asking with an - extension in arm's reach and must still not see it. + ``before`` is one directory below the ``extensions`` directory this session + has, which is as close as an adopter with none can get to having one. It runs + the canonical set and nothing else, so the directory a module is in -- not + the session it runs in -- is what decides what it collects. """ assert (adoption.directory / EXTENSIONS_DIRECTORY).is_dir() + assert not (adoption.directory / BASELINE_DIRECTORY / EXTENSIONS_DIRECTORY).exists() assert set(_of(adoption, "test_before")) < set(_of(adoption, "test_after")) @@ -357,7 +388,7 @@ def test_feature_paths_is_the_canonical_set_when_there_is_no_extension_directory ): """This test module has no ``extensions`` beside it, and gets one path.""" assert not (Path(__file__).parent / EXTENSIONS_DIRECTORY).exists() - assert feature_paths() == (features_path(),) + assert [Path(path).resolve() for path in feature_paths()] == [CANONICAL_ROOT] def test_an_extension_directory_counts_only_when_it_is_a_directory( @@ -381,7 +412,7 @@ def test_an_extension_directory_counts_only_when_it_is_a_directory( def test_the_canonical_features_are_found_inside_the_distribution() -> None: """No submodule and no directory layout of the adopter's own.""" - packaged = Path(features_path()) + packaged = CANONICAL_ROOT assert (packaged / CANONICAL_FEATURE).is_file() assert is_canonical(packaged / CANONICAL_FEATURE) assert not is_canonical(Path(__file__)) @@ -409,7 +440,7 @@ def test_the_two_prefixes_are_the_ones_appendix_f_names() -> None: def test_the_canonical_assets_keep_the_reserved_prefix() -> None: - canonical = Path(features_path()) / CANONICAL_FEATURE + canonical = CANONICAL_ROOT / CANONICAL_FEATURE assert uri_for(canonical) == f"{CANONICAL_DIRECTORY}/{CANONICAL_FEATURE}" assert is_canonical_uri(f"{CANONICAL_DIRECTORY}/{CANONICAL_FEATURE}") assert ( diff --git a/tools/openfeature-tck/tests/test_in_memory_conformance.py b/tools/openfeature-tck/tests/test_in_memory_conformance.py index b4a256e3d..1de4018ad 100644 --- a/tools/openfeature-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-tck/tests/test_in_memory_conformance.py @@ -24,7 +24,7 @@ Capability, TckConfig, canonical_flag_set, - features_path, + feature_paths, ) from openfeature.provider import FeatureProvider from openfeature.provider.in_memory_provider import InMemoryProvider @@ -144,4 +144,4 @@ def tck_config() -> TckConfig: ) -scenarios(features_path()) +scenarios(*feature_paths()) From 25018b87099aa0d24fdfbaaeca42da20d14bcb56 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 14:06:06 +0200 Subject: [PATCH 25/46] feat(tck)!: an untracked deviation, and a summary that is really required `KnownDeviation` was the one place where this suite's contract differed from the other three languages': `issue` was required and there was no untracked form, so Python alone could not record a defect that is real but not filed anywhere. It gains `KnownDeviation.tracked()` and `KnownDeviation.untracked()`, the same pair Go, Java and JavaScript have, and the constructor's field order changes accordingly -- `summary` first, `issue` optional after it. The docstring on `issue` said "A URI, because the schema requires one." That was false, and it had been false since the schema existed. `$defs.knownDeviation` in conformance-report.schema.json carried no `required` array at all at spec 4079ce0f, so neither field was required; spec fcd63415 has since added `"required": ["summary"]`, which requires `summary` and still not `issue`. Either way the claim was backwards about the one field it named. Removed. `summary` is now genuinely required and validated, with the reason in the message: a deviation with no summary records that something is wrong without saying what, which leaves a reader worse off than the bare skip or failure it accompanies. A deviation naming a reserved capability is refused for the same reason declaring one is -- no scenario carries the tag, so nothing was failed or skipped for the deviation to explain. The `capability` docstring said a deviation against a mandatory scenario was "the common case, since a capability a provider fails is usually one it should not have declared". That states the withheld-capability shape as the default and contradicts the settled guidance, which prefers the other one: declare the capability, let the scenario fail, and record the deviation beside the visible failure. Both shapes are now spelled out, with the preference and with the reason the second one is narrow -- withdrawing a capability in order to turn a failure into a skip is the failure mode the field exists to prevent. `as_json()` omits `issue` rather than emitting null, because the schema types it as a uri-formatted string when present. test_declaration.py also swaps `features_path()` for `canonical_root()`, which belongs to the preceding commit and lands here because it is the same file. Signed-off-by: Simon Schrottner --- .../openfeature/contrib/tools/tck/config.py | 158 ++++++++++++++++-- .../openfeature-tck/tests/test_declaration.py | 88 ++++++++-- 2 files changed, 220 insertions(+), 26 deletions(-) diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py index 6ff454584..13ba53f91 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py @@ -29,36 +29,114 @@ class KnownDeviation: """A gap the provider is known to have, acknowledged rather than hidden. - Distinct from an undeclared capability, which is a choice the provider is - entitled to make: this is a defect against something the specification does - not treat as optional, with the gap tracked somewhere. + **A ``knownDeviations`` entry says: this provider fails to do something it is + required to do.** The requirement must be a numbered ``MUST``, or a rule the + implementation bound itself to elsewhere. Distinct from an undeclared + capability, which is a *choice* the provider is entitled to make: where the + specification permits the choice, withholding the capability **is** the + honest report, and a deviation entry would assert a defect that does not + exist. + + It is legitimate in two shapes, and a report's results already distinguish + them: + + 1. **The capability is declared, the scenario runs, and it fails.** Prefer + this. The failure stays visible and the deviation says it is known and + why. + 2. **The capability is withheld, and its scenarios skip.** Legitimate only + when the provider cannot attempt the behaviour at all, so running the + scenario would establish nothing. The deviation then explains the + absence, so a reader can tell a defect from a design decision. + + Withdrawing a capability *in order to* turn a failing scenario into a skip is + the failure mode this field exists to prevent. If the provider attempts the + behaviour and gets it wrong, shape 1 is the honest report. It changes nothing about how the suite runs. The scenario still fails, and the results payload still reports it as failed -- a report that softened a failure into a footnote would hide exactly what the acknowledgement exists to keep visible. What this adds is the acknowledgement itself, in the envelope, - so that a consumer can tell a known and tracked gap from a surprise. - """ + so that a consumer can tell a known gap from a surprise. - issue: str - """Where the gap is tracked. A URI, because the schema requires one.""" + Build one with :meth:`tracked` or :meth:`untracked` rather than by calling + the constructor, so that which of the two a deviation is stays a decision + someone made rather than a field someone forgot. The same two forms exist in + the Go, Java and JavaScript suites. + """ summary: str - """What is wrong, for a person reading a comparison page.""" + """What is wrong, for a person reading a comparison page. + + Required. A deviation with no summary records that something is wrong without + saying what, which is worth less than the bare skip or failure it + accompanies. + """ + + issue: str | None = None + """Where the gap is tracked, or ``None`` when it is tracked nowhere yet. + + Optional. There is a tracked and an untracked form, and naming an untracked + defect is still what separates it from a capability the provider chose to + withhold -- a declaration that merely omits the tag cannot say which of the + two happened. Prefer :meth:`tracked` as soon as there is an issue to point + at. + """ capability: Capability | None = None """The capability the deviation concerns, when it maps to one. - Left out for a deviation against a mandatory scenario, which belongs to no - capability -- which is the common case, since a capability a provider fails - is usually one it should not have declared. + Left out when the gap is against a mandatory, ungated scenario, which belongs + to no capability. + + A reserved capability is refused: no scenario carries the tag, so there is + nothing to deviate from. See :data:`~.capability.RESERVED_CAPABILITIES`. """ + @classmethod + def tracked( + cls, + summary: str, + issue: str, + capability: Capability | None = None, + ) -> KnownDeviation: + """Record a deviation that is tracked somewhere. + + :param summary: what the gap is. + :param issue: a URI where it is tracked. + :param capability: the capability the gap concerns, or ``None`` when the + gap is against a mandatory scenario and so belongs to no capability. + """ + return cls(summary=summary, issue=issue, capability=capability) + + @classmethod + def untracked( + cls, + summary: str, + capability: Capability | None = None, + ) -> KnownDeviation: + """Record a deviation that is not tracked anywhere yet. + + Worth declaring even so: naming the defect is what separates it from a + capability the provider chose to withhold. Prefer :meth:`tracked` as soon + as there is an issue to point at. + + :param summary: what the gap is. + :param capability: the capability the gap concerns, or ``None`` when the + gap is against a mandatory scenario and so belongs to no capability. + """ + return cls(summary=summary, capability=capability) + + @property + def is_tracked(self) -> bool: + """Whether this deviation points at somewhere the gap is tracked.""" + return bool(self.issue) + def as_json(self) -> dict[str, typing.Any]: - document: dict[str, typing.Any] = { - "issue": self.issue, - "summary": self.summary, - } + document: dict[str, typing.Any] = {"summary": self.summary} + # Omitted rather than null: the schema's `issue` is a uri-formatted + # string when present, so an untracked deviation leaves the key out. + if self.issue is not None: + document["issue"] = self.issue if self.capability is not None: document["capability"] = self.capability.tag return document @@ -205,6 +283,7 @@ def __post_init__(self) -> None: object.__setattr__(self, "known_deviations", tuple(self.known_deviations)) problems.extend(reserved_problems(self.capabilities)) + problems.extend(deviation_problems(self.known_deviations)) if ( Capability.UNAVAILABLE_INIT in self.capabilities @@ -274,6 +353,55 @@ def reserved_problems(declared: Iterable[Capability]) -> list[str]: ] +def deviation_problems(deviations: Sequence[KnownDeviation]) -> list[str]: + """Refuse a deviation that says nothing a consumer can use. + + The rules are deliberately narrow. A deviation is prose written by the + provider author for a human comparing providers, and no suite can check + prose; what it can check is that the prose is there and that the capability + it names is one a scenario could have been gated on. + + A reserved capability is refused for the same reason declaring one is: no + scenario carries the tag, so there is no failure and no skip for the + deviation to explain, and nothing it could be about. + """ + problems: list[str] = [] + + for index, deviation in enumerate(deviations): + if not deviation.summary or not deviation.summary.strip(): + problems.append( + f"known_deviations[{index}] has no summary: a deviation exists to " + f"say what the gap is, and one that does not say it leaves a " + f"consumer no better off than the bare skip or failure it " + f"accompanies. It is the one field the report schema requires" + ) + + capability = deviation.capability + if capability is None: + # Legitimate: the gap is against a mandatory, ungated scenario, + # which belongs to no capability. + continue + + if not isinstance(capability, Capability): + problems.append( + f"known_deviations[{index}] names unknown capability " + f"{capability!r}: capabilities are the members of the Capability " + f"enum" + ) + continue + + if capability.reserved: + problems.append( + f"known_deviations[{index}] names the reserved capability " + f"{capability.tag}: no scenario carries that tag, so nothing was " + f"failed or skipped for this deviation to explain and no result " + f"could show the gap. Remove it, or name the capability whose " + f"scenarios the gap actually affects" + ) + + return problems + + def capabilities_of(values: Iterable[Capability]) -> frozenset[Capability]: """Convenience for building a capability set from any iterable.""" return frozenset(values) diff --git a/tools/openfeature-tck/tests/test_declaration.py b/tools/openfeature-tck/tests/test_declaration.py index 8af5eb9a5..ba8d6a112 100644 --- a/tools/openfeature-tck/tests/test_declaration.py +++ b/tools/openfeature-tck/tests/test_declaration.py @@ -23,7 +23,6 @@ import re import typing -from pathlib import Path import pytest @@ -35,7 +34,7 @@ InProcessControl, KnownDeviation, TckConfig, - features_path, + canonical_root, ) from openfeature.contrib.tools.tck.capability import ( capability_for_marker, @@ -80,7 +79,9 @@ def _canonical_tags() -> set[str]: those scenarios will go once they exist. """ tags: set[str] = set() - for feature in sorted(Path(features_path()).glob("*.feature")): + root = canonical_root() + assert root is not None, "the packaged canonical features are not on a filesystem" + for feature in sorted(root.glob("*.feature")): for line in feature.read_text(encoding="utf-8").splitlines(): stripped = line.strip() if stripped.startswith("@"): @@ -229,28 +230,91 @@ def test_the_refusal_says_what_may_be_declared_instead() -> None: # -- acknowledging a gap ----------------------------------------------------- +ISSUE = "https://github.com/open-feature/python-sdk/issues/619" + + def test_a_known_deviation_carries_its_capability_only_when_it_has_one() -> None: - """The common case has none: a mandatory scenario belongs to no capability. + """A deviation against a mandatory, ungated scenario belongs to no capability. Omitted rather than null, because the field is the answer to "which capability does this concern" and there is not always one. """ - issue = "https://github.com/open-feature/python-sdk/issues/619" - mandatory = KnownDeviation(issue=issue, summary="a boolean satisfies an Integer") - assert mandatory.as_json() == {"issue": issue, "summary": mandatory.summary} + mandatory = KnownDeviation.tracked( + summary="a boolean satisfies an Integer", issue=ISSUE + ) + assert mandatory.as_json() == {"issue": ISSUE, "summary": mandatory.summary} - attributed = KnownDeviation( - issue=issue, + attributed = KnownDeviation.tracked( summary="a lossy float satisfies an Integer", + issue=ISSUE, capability=Capability.NUMERIC_COERCION, ) assert attributed.as_json() == { - "issue": issue, + "issue": ISSUE, "summary": attributed.summary, "capability": Capability.NUMERIC_COERCION.tag, } +def test_an_untracked_deviation_is_a_form_of_its_own() -> None: + """Because a gap with nowhere to point at is still worth naming. + + Naming the defect is what separates it from a capability the provider chose + to withhold; a declaration that merely omits the tag cannot say which of the + two happened. This suite had no way to record one until now -- ``issue`` was + required -- and was the only one of the four that had not. + + ``issue`` is left out of the payload rather than sent as null: the report + schema types it as a uri-formatted string when present, and requires only + ``summary``. + """ + untracked = KnownDeviation.untracked( + summary="the lossy half of the coercion rule is not enforced", + capability=Capability.NUMERIC_COERCION, + ) + + assert not untracked.is_tracked + assert untracked.issue is None + assert untracked.as_json() == { + "summary": untracked.summary, + "capability": Capability.NUMERIC_COERCION.tag, + } + + assert KnownDeviation.tracked(summary="a gap", issue=ISSUE).is_tracked + + +def test_a_deviation_with_no_summary_is_refused() -> None: + """It records that something is wrong without saying what. + + Which leaves a reader worse off than the bare skip or failure it + accompanies, and it is the one field the report schema requires. Refused at + construction, where the adopter's own code is still on the stack to say + which line to fix. + """ + with pytest.raises(ValueError, match="known_deviations\\[0\\] has no summary"): + _config(known_deviations=[KnownDeviation.untracked(summary=" ")]) + + +def test_a_deviation_may_not_name_a_reserved_capability() -> None: + """No scenario carries the tag, so there is nothing to deviate from. + + The same reason declaring one is refused: there is no failure and no skip + for the deviation to explain, so the entry would tell a reader only that + something was claimed about something nothing examined. + """ + reserved = next(iter(RESERVED_CAPABILITIES)) + with pytest.raises(ValueError) as raised: + _config( + known_deviations=[ + KnownDeviation.untracked(summary="a gap", capability=reserved) + ] + ) + + message = str(raised.value) + assert f"names the reserved capability {reserved.tag}" in message + assert "nothing was failed or skipped" in message + + def test_known_deviations_are_normalised_and_change_nothing_about_the_run() -> None: """Declared as any sequence; read as a tuple. @@ -258,7 +322,9 @@ def test_known_deviations_are_normalised_and_change_nothing_about_the_run() -> N nothing here makes a scenario pass, skip, or be collected differently, which is why a suite declaring one still fails on it. """ - deviation = KnownDeviation(issue="https://example.invalid/1", summary="a gap") + deviation = KnownDeviation.tracked( + summary="a gap", issue="https://example.invalid/1" + ) config = _config(known_deviations=[deviation]) assert config.known_deviations == (deviation,) assert _config().known_deviations == () From 390c6fd58167d722f51e3035894ee039425def67 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 14:06:26 +0200 Subject: [PATCH 26/46] feat(tck): the suite owns the container stack, not every adopter A provider that talks to a backend needs that backend running, its dynamically mapped host ports discovered, and an `HttpControl` built against its control API. Every adoption needs the same three things and until now every adoption wrote them: this package shipped the control client and left orchestration to the adopter, so the flagd adoption alone carried a 122-line `conftest.py` and a 170-line `suite.py` of container wiring, and the next adopter would have paid for it again. It is the single largest adoption cost in three of the four languages. An adopter now names a Compose file, says which ports the provider connects to, and builds a provider from an endpoint they are handed: @pytest.fixture(scope="session") def compose_backend() -> ComposeBackend: return ComposeBackend( compose_file="tests/tck/docker-compose.yaml", backend_ports=[8013], ) @pytest.fixture(scope="session") def tck_config(tck_backend: RunningBackend) -> TckConfig: return TckConfig( name="my-provider", control=tck_backend.control, new_provider=lambda: MyProvider( host=tck_backend.endpoint.host, port=tck_backend.endpoint.port(8013), ), ) `ComposeBackend` carries the concepts and defaults Java's `ContainerizedProviderTckTest` fixed -- backend service `backend`, control port 8080, config `default`, startup timeout 60s, plus `additional_ports` for a stack with more than one service -- so a provider shipped in two languages writes one Compose file and two declarations against it. `tck_backend` is a session-scoped fixture of this package's plugin, and it is lazy: an in-memory adopter never requests it, never defines `compose_backend`, and never needs Docker. The stack starts once per suite and is never restarted, because orchestrators cannot reliably preserve dynamically mapped host ports across a restart and a restart would silently invalidate every provider already pointed at the old one. Unavailability stays simulated inside the running stack through the control API. The Compose path is additional rather than a replacement: a provider with no backend keeps supplying its own `BackendControl`. Startup is a readiness check rather than a pause. `docker compose up --wait` brings the containers up, every declared port is then waited on until it accepts a connection -- the guarantee Java gets from a Testcontainers listening-port wait strategy, which `--wait` alone does not give for a service with no healthcheck -- and `HttpControl.await_ready()` probes `GET /healthz` until the control API answers, treating 404 as ready because `control-api.yaml` defines it that way and the reference launchpad serves no such path. There is deliberately no settle *after* a control call. Java sleeps 50ms after every one; flagd-testbed#394 makes `POST /start` block until the flags are evaluable, so the sleep covers a window that no longer exists, and a suite that sleeps instead of holding the control API to its promise stops being able to detect when the promise breaks. `testcontainers` is the optional `compose` extra rather than a dependency, imported lazily, so an in-memory adopter does not install container tooling to run a suite that never starts a container. It is in the dev group so the lazy import is type-checked, with a mypy override because testcontainers 4.14 ships no py.typed for `testcontainers.compose`. The port resolution is checked against a `ComposeStack` protocol instead, which is what lets it be tested without Docker at all -- the stack lifecycle is proved by the flagd adoption. `__init__.py` and the README also carry the export list and the documentation for the two preceding commits, because the public surface and the docs move together. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 156 +++++- tools/openfeature-tck/pyproject.toml | 22 + .../openfeature/contrib/tools/tck/__init__.py | 57 +- .../openfeature/contrib/tools/tck/compose.py | 487 ++++++++++++++++++ .../contrib/tools/tck/httpcontrol.py | 85 ++- .../openfeature/contrib/tools/tck/plugin.py | 32 ++ tools/openfeature-tck/tests/test_compose.py | 352 +++++++++++++ .../tests/test_http_control.py | 97 +++- uv.lock | 9 + 9 files changed, 1268 insertions(+), 29 deletions(-) create mode 100644 tools/openfeature-tck/src/openfeature/contrib/tools/tck/compose.py create mode 100644 tools/openfeature-tck/tests/test_compose.py diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 2dfc989dc..fff3c84f8 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -26,7 +26,7 @@ mechanism once, not exhaustive coverage. Breaking changes should be expected. ## Adopting it -One fixture and one call. It uses **pytest-bdd**, the same runner the flagd provider and the flagd +Two fixtures and one call. It uses **pytest-bdd**, the same runner the flagd provider and the flagd testkit already use, so an adopting package gains no new test framework. ```python @@ -35,18 +35,30 @@ from pytest_bdd import scenarios from openfeature.contrib.tools.tck import ( Capability, + ComposeBackend, + RunningBackend, TckConfig, feature_paths, ) @pytest.fixture(scope="session") -def tck_config(): - control = MyBackendControl() +def compose_backend(): + return ComposeBackend( + compose_file="tests/tck/docker-compose.yaml", + backend_ports=[8013], + ) + + +@pytest.fixture(scope="session") +def tck_config(tck_backend: RunningBackend): return TckConfig( name="my-provider", - control=control, - new_provider=lambda: MyProvider(control.address), + control=tck_backend.control, + new_provider=lambda: MyProvider( + host=tck_backend.endpoint.host, + port=tck_backend.endpoint.port(8013), + ), capabilities={Capability.EVENTS, Capability.OBJECT}, ) @@ -54,6 +66,14 @@ def tck_config(): scenarios(*feature_paths()) ``` +**The suite owns the container stack.** You name a Compose file, say which ports the provider +connects to, and build a provider from the endpoint you are handed. Starting the stack, discovering +the dynamically mapped host ports, building the HTTP control against the control API, waiting until +it accepts commands and tearing down afterwards are all the suite's - see +[The container stack](#the-container-stack). A provider with **no** backend supplies a +`BackendControl` of its own instead and needs no Compose file and no container tooling - see +[Providers with no backend](#providers-with-no-backend). + There is **no `conftest.py` to write and nothing to import for the steps**. The step definitions arrive through this package's pytest plugin, registered via a `pytest11` entry point, so installing the package is all it takes. @@ -116,9 +136,16 @@ inside the installed distribution rather than in your repository. `feature_paths scenarios(*feature_paths()) ``` -That line does not change when you add an extension, and it is the only difference from -`scenarios(features_path())` — which still works and still sees only the canonical set. An adopter -with no `extensions` directory runs exactly what they ran before: same scenarios, same count. +That line does not change when you add an extension. An adopter with no `extensions` directory gets +the canonical set alone, so adding one is a matter of creating a directory rather than of +configuring anything. + +There used to be a second call, `features_path()`, which returned the canonical set on its own. It +is **gone**. The two differed by one character at the call site and the shorter one silently dropped +the extensions directory, so reaching for it produced a green run over fewer scenarios than the +adopter believed had run — which is the worst failure mode available to a conformance suite, because +nothing is there to notice. `canonical_root()` is the supported way to reach the packaged directory +for anything that is not "the scenarios to run". ### Your scenarios cannot stand in for ours @@ -335,10 +362,43 @@ up on and fails its scenario with a message rather than hanging the session. One further field on `TckConfig` says something a capability set cannot, and it is a declaration rather than a switch: it changes neither which scenarios run nor what they assert. -`known_deviations=(KnownDeviation(issue=..., summary=...),)` acknowledges a gap against something the -specification does *not* treat as optional, with somewhere it is tracked. It is an acknowledgement -and not an excuse: the scenario still fails and the suite still fails with it. What the declaration -adds is that the gap was known rather than a surprise. +`known_deviations` acknowledges a gap against something the specification does *not* treat as +optional. It is an acknowledgement and not an excuse: the scenario still fails and the suite still +fails with it. What the declaration adds is that the gap was known rather than a surprise. + +```python +TckConfig( + # ... + known_deviations=( + KnownDeviation.tracked( + summary="what is wrong, for someone comparing providers", + issue="https://github.com/open-feature/flagd/issues/1996", + capability=Capability.NUMERIC_COERCION, + ), + ), +) +``` + +- **`summary` is required.** A deviation with no summary records that something is wrong without + saying what, which leaves a reader worse off than the bare skip or failure it accompanies. +- **`issue` is optional**, and `KnownDeviation.untracked(summary=...)` is the form for a gap that is + not tracked anywhere yet. Naming an untracked defect is still what separates it from a capability + the provider chose to withhold; prefer the tracked form as soon as there is somewhere to point. +- **`capability` is optional**, and left out when the gap is against a mandatory, ungated scenario. + A reserved capability is refused: no scenario carries the tag, so there is nothing to deviate + from. + +It is legitimate in two shapes, and **prefer the first**: + +1. **The capability is declared, the scenario runs, and it fails.** The failure stays visible and + the deviation says it is known and why. +2. **The capability is withheld and its scenarios skip.** Legitimate only when the provider cannot + attempt the behaviour at all, so running the scenario would establish nothing. The deviation then + explains the absence, so a reader can tell a defect from a design decision. + +Withdrawing a capability *in order to* turn a failing scenario into a skip is the failure mode this +field exists to prevent. Where the specification permits the choice, withholding the capability +**is** the honest report and a deviation entry would assert a defect that does not exist. ## Controlling the backend @@ -355,11 +415,70 @@ TCK drives the same endpoints against the same stack and must get the same answe control = HttpControl(f"http://localhost:{container.get_launchpad_port()}") ``` -`HttpControl` is built on `urllib.request` alone, so the TCK gains no HTTP client and no container -dependency. **Orchestrating the stack stays with you**, where the vendor-specific knowledge already -lives — which compose file, which services, which internal ports. That is a deliberate trade against -the "provider authors write no test infrastructure" goal, and worth revisiting once a second -containerised adopter shows what is actually common. +`HttpControl` is built on `urllib.request` alone, so the TCK gains no HTTP client. You do not build +one yourself when you use the Compose harness below: it is handed to you as `tck_backend.control`, +already awaited ready. + +### The container stack + +The suite starts it. An adopter used to write the container wrapper — and every adopter wrote the +same one, which is why the flagd adoption alone carried a 122-line `conftest.py` and a 170-line +`suite.py` of it. `ComposeBackend` is the whole declaration: + +| field | required | default | meaning | +| --- | --- | --- | --- | +| `compose_file` | yes | — | path to the Compose file, resolved relative to the package directory | +| `backend_ports` | yes | — | container-internal ports the **provider** connects to. The control port is exposed automatically and must not be listed here | +| `backend_service` | no | `"backend"` | the Compose service hosting both the control API and the backend | +| `control_port` | no | `8080` | container-internal port of the control API | +| `additional_ports` | no | `{}` | extra service to ports, for a stack with more than one service. Resolved through the endpoint by service name | +| `configuration` | no | `"default"` | the configuration name passed to `POST /start` | +| `startup_timeout` | no | `60.0` | seconds to wait for the stack and its control API to become reachable | + +Those names and defaults are fixed across all four languages' TCKs, so a provider shipped in two of +them writes one Compose file and two declarations against it. + +`tck_backend` is a session-scoped fixture this package's plugin supplies, and it yields two things: + +- `tck_backend.control` — the `HttpControl`, already awaited ready. Hand it to `TckConfig.control`. + One per stack: it remembers whether a disconnect left the backend down, so two suites driving the + same backend must share it. +- `tck_backend.endpoint` — `host`, `port(internal)` and `port(internal, service=...)`. This is a + **factory argument, not a field**: the mapped ports do not exist until the stack is up, which is + why `TckConfig.new_provider` is a factory called once per scenario. + +Your Compose file must **not pin host ports**. Docker assigns them dynamically and the harness +discovers them after startup; a pinned host port makes the suite unrunnable in parallel and collides +with whatever you already have listening. Declaring `backend_ports` is what lets the harness say +"the Compose file does not publish 8013" at startup rather than leaving you with a provider that +cannot connect three scenarios later. + +Startup is a real readiness check rather than a pause: the stack comes up with +`docker compose up --wait`, then every declared port is waited on until it accepts a connection, +then `HttpControl.await_ready()` probes `GET /healthz` until the control API answers. There is +deliberately **no settle after a control call**. Java had a fixed 50ms one; flagd-testbed#394 makes +`POST /start` block until the flags are evaluable, so the sleep covered a window that no longer +exists — and a suite that sleeps instead of holding the control API to its promise stops being able +to detect when the promise breaks. If dropping it makes an adoption flaky, that is a testbed defect +worth filing, not a sleep worth restoring. + +`testcontainers` is an **optional** extra rather than a dependency: + +``` +pip install 'openfeature-tck[compose]' +``` + +An in-memory adopter should not have to install container tooling to run a suite that never starts a +container, so `compose.py` imports it lazily and says so if it is missing. + +If you want the fixture under a different name or scope, `run_compose_backend()` is the generator +behind it: + +```python +@pytest.fixture(scope="session") +def tck_backend(): + yield from run_compose_backend(ComposeBackend(...)) +``` Two of its behaviours are worth knowing about: @@ -521,9 +640,6 @@ both. Both declare `@variants`, since an in-memory flag set is keyed by variant *whole* context arrives intact: a provider that forwards the targeting key and silently discards every other attribute passes. That needs either an echo operation on the control API or a second canonical flag whose rule keys on a custom attribute. -- **No shared containerised-backend helper.** `HttpControl` drives the control API, but starting the - stack and discovering its mapped ports is still each adopter's own code. Abstracting that from a - single example tends to produce the wrong abstraction; it should wait for a second adopter. - **Caching, hooks and flag metadata** are not covered. [appendix-a]: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md diff --git a/tools/openfeature-tck/pyproject.toml b/tools/openfeature-tck/pyproject.toml index 28886de94..efc89fbe5 100644 --- a/tools/openfeature-tck/pyproject.toml +++ b/tools/openfeature-tck/pyproject.toml @@ -25,6 +25,12 @@ dependencies = [ ] requires-python = ">=3.10" +# Kept out of the required dependencies on purpose: a provider with no backend +# runs the whole suite without a container, and should not have to install +# container tooling to do it. Everything in `compose.py` imports it lazily. +[project.optional-dependencies] +compose = ["testcontainers>=4.12.0,<5.0.0"] + [project.urls] Homepage = "https://github.com/open-feature/python-sdk-contrib" @@ -40,6 +46,11 @@ dev = [ "coverage[toml]>=7.10.0,<8.0.0", "mypy>=1.18.0,<2.0.0", "poethepoet>=0.37.0", + # The `compose` extra, so the lazy import in compose.py is type-checked + # against the real DockerCompose rather than waved through. Nothing in this + # package's own tests starts a container -- the flagd adoption is what proves + # the harness end to end. + "testcontainers>=4.12.0,<5.0.0", ] [tool.hatch.build.targets.sdist] @@ -75,6 +86,17 @@ pretty = true strict = true disallow_any_generics = false +# testcontainers 4.14 ships no py.typed for `testcontainers.compose`, so the one +# lazy import in compose.py cannot be checked against it. Scoped to that module +# rather than made a global `ignore_missing_imports`, and it costs nothing here: +# the harness talks to a stack through the `ComposeStack` protocol, which is what +# the port resolution is actually checked against. +[[tool.mypy.overrides]] +module = [ + "testcontainers.*", +] +ignore_missing_imports = true + [tool.coverage.run] omit = ["tests/**"] diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py index 24d279e59..c20926ba0 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py @@ -7,18 +7,49 @@ shared basis is the whole point -- "conformant" only means something if the question is identical everywhere. -**What a provider author writes.** One fixture and one call:: +**What a provider author writes.** Two fixtures and one call:: import pytest from pytest_bdd import scenarios from openfeature.contrib.tools.tck import ( Capability, - InProcessControl, + ComposeBackend, + RunningBackend, TckConfig, feature_paths, ) + @pytest.fixture(scope="session") + def compose_backend(): + return ComposeBackend( + compose_file="tests/tck/docker-compose.yaml", + backend_ports=[8013], + ) + + @pytest.fixture(scope="session") + def tck_config(tck_backend: RunningBackend): + return TckConfig( + name="my-provider", + control=tck_backend.control, + new_provider=lambda: MyProvider( + host=tck_backend.endpoint.host, + port=tck_backend.endpoint.port(8013), + ), + capabilities={Capability.EVENTS, Capability.OBJECT}, + ) + + scenarios(*feature_paths()) + +The suite owns the container stack: it starts the Compose file once, discovers +the dynamically mapped host ports, builds the HTTP control against the control +API, waits until it accepts commands, and tears down after the last scenario. +See :mod:`~.compose`. + +**A provider with no backend supplies its own control instead** -- in-memory, +environment-variable, file-based -- and needs no Compose file and no container +tooling:: + @pytest.fixture(scope="session") def tck_config(): control = InProcessControl() @@ -29,8 +60,6 @@ def tck_config(): capabilities={Capability.EVENTS, Capability.OBJECT}, ) - scenarios(*feature_paths()) - ``scenarios()`` is pytest-bdd's own, called directly rather than wrapped: it injects the generated tests into the *calling module* by walking the stack, so a convenience wrapper around it would deposit them inside this package instead. @@ -53,6 +82,14 @@ def tck_config(): import importlib.resources from .capability import DECLARABLE_CAPABILITIES, RESERVED_CAPABILITIES, Capability +from .compose import ( + DEFAULT_BACKEND_SERVICE, + DEFAULT_CONTROL_PORT, + BackendEndpoint, + ComposeBackend, + RunningBackend, + run_compose_backend, +) from .config import KnownDeviation, TckConfig from .control import ( BackendControl, @@ -61,11 +98,12 @@ def tck_config(): ) from .extensions import ( EXTENSIONS_DIRECTORY, + canonical_root, feature_paths, - features_path, ) from .httpcontrol import ( DEFAULT_CONFIGURATION, + DEFAULT_STARTUP_TIMEOUT, ControlApiError, HttpControl, ) @@ -81,25 +119,32 @@ def tck_config(): __all__ = [ "CHANGING_FLAG_KEY", "DECLARABLE_CAPABILITIES", + "DEFAULT_BACKEND_SERVICE", "DEFAULT_CONFIGURATION", + "DEFAULT_CONTROL_PORT", + "DEFAULT_STARTUP_TIMEOUT", "EXTENSIONS_DIRECTORY", "RESERVED_CAPABILITIES", "BackendControl", + "BackendEndpoint", "Capability", + "ComposeBackend", "ConnectionControl", "ControlApiError", "ControllableInMemoryProvider", "HttpControl", "InProcessControl", "KnownDeviation", + "RunningBackend", "TckConfig", "TckState", "UnsupportedControlError", "canonical_flag_set", "canonical_flags_json", + "canonical_root", "control_api_spec", "feature_paths", - "features_path", + "run_compose_backend", ] # NOTE ON THE SOURCE OF TRUTH diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/compose.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/compose.py new file mode 100644 index 000000000..7e7f36397 --- /dev/null +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/compose.py @@ -0,0 +1,487 @@ +"""The container stack, owned by the suite rather than by every adopter. + +A provider that talks to a backend needs that backend running, its dynamically +mapped host ports discovered, and an :class:`~.httpcontrol.HttpControl` built +against its control API. Every adoption needs the same three things, and until +now every adoption wrote them: this package shipped the control client and left +orchestration to the adopter, so the flagd adoption alone carried a 122-line +``conftest.py`` and a 170-line ``suite.py`` of container wiring, and the next +adopter would have paid for it again. + +So the suite owns the stack. **An adopter names a Compose file, says which +service and ports to expose, and supplies a factory that builds a provider from +a discovered endpoint.** Everything else -- start the stack once, discover the +mapped ports, build the control, wait until it accepts commands, tear down after +the last scenario -- happens here. + +Two lines, in the adopter's ``conftest.py``:: + + @pytest.fixture(scope="session") + def compose_backend() -> ComposeBackend: + return ComposeBackend( + compose_file="tests/tck/docker-compose.yaml", + backend_ports=[8013], + ) + + @pytest.fixture(scope="session") + def tck_config(tck_backend: RunningBackend) -> TckConfig: + return TckConfig( + name="my-provider", + control=tck_backend.control, + new_provider=lambda: MyProvider( + host=tck_backend.endpoint.host, + port=tck_backend.endpoint.port(8013), + ), + ) + +``tck_backend`` is a session-scoped fixture this package's plugin supplies. It +is lazy, so a provider with no backend never touches it and never needs Docker +installed -- see :mod:`~.inprocess`. + +**The stack starts once per suite and is never restarted.** Container +orchestrators assign host ports dynamically and cannot reliably preserve them +across a restart, so a restarted backend comes back on a different host port, +silently invalidating every provider already pointed at the old one -- and the +failure looks like a flaky provider rather than a broken test. Backend +unavailability is always simulated *inside* the running stack, through the +control API. That is also why :attr:`TckConfig.new_provider` is a factory rather +than an instance: the ports do not exist until the stack is up. + +**The Compose path does not replace the manual one.** A provider with no backend +at all keeps supplying its own :class:`~.control.BackendControl` exactly as +before. Compose is an additional path, and the one nearly every provider wants. + +**Do not substitute a control of your own that pokes an external backend through +a side channel.** The HTTP control API is the normative contract: another +language's suite drives the same endpoints against the same stack and must get +the same answers. See :class:`~.control.BackendControl`. + +Requires the ``compose`` extra -- ``pip install 'openfeature-tck[compose]'`` -- +which is what pulls in ``testcontainers``. Keeping it optional is deliberate: an +in-memory adopter should not have to install container tooling to run a suite +that never starts a container. +""" + +from __future__ import annotations + +import contextlib +import socket +import time +import typing +from collections.abc import Iterator, Mapping, Sequence +from dataclasses import dataclass, field +from pathlib import Path + +from .httpcontrol import ( + DEFAULT_CONFIGURATION, + DEFAULT_STARTUP_TIMEOUT, + ControlApiError, + HttpControl, +) + +__all__ = [ + "DEFAULT_BACKEND_SERVICE", + "DEFAULT_CONTROL_PORT", + "BackendEndpoint", + "ComposeBackend", + "ComposeStack", + "RunningBackend", + "run_compose_backend", +] + +DEFAULT_BACKEND_SERVICE = "backend" +"""The Compose service name a stack is expected to host the backend under. + +Fixed across every language's TCK, so the same Compose file is the whole of what +an adoption shares between two of them. +""" + +DEFAULT_CONTROL_PORT = 8080 +"""The container-internal port the control API is expected to listen on.""" + +_PORT_PROBE_TIMEOUT = 1.0 +"""Seconds bounding a single TCP connect while waiting for a published port.""" + +_PORT_POLL_INTERVAL = 0.2 + + +class ComposeStack(typing.Protocol): + """The two things this module needs from a Compose stack. + + A protocol rather than ``testcontainers.compose.DockerCompose`` itself, so + that port resolution is checkable and testable without Docker installed and + without the optional dependency present. The real class satisfies it + structurally. + """ + + def get_service_host( + self, service_name: str | None = ..., port: int | None = ... + ) -> str | None: ... + + def get_service_port( + self, service_name: str | None = ..., port: int | None = ... + ) -> int | None: ... + + +@dataclass(frozen=True) +class BackendEndpoint: + """Where the running stack is reachable, handed to the provider factory. + + This type exists because host ports are only known *after* the stack has + started. A Compose file under test must not pin host ports -- Docker assigns + them dynamically -- so a provider cannot be configured until the stack is up, + which is the whole reason :attr:`TckConfig.new_provider` is a factory. + + The mapping is stable for the lifetime of the suite: the stack is started + once and never restarted, so a provider built from this endpoint stays valid + for every scenario. + """ + + stack: ComposeStack + """The running stack the addresses are resolved from.""" + + backend_service: str = DEFAULT_BACKEND_SERVICE + """The service :attr:`host` and the one-argument :meth:`port` resolve against.""" + + service_ports: Mapping[str, Sequence[int]] = field(default_factory=dict) + """Which container-internal ports each service publishes, from the declaration. + + Carried so that a host can be resolved at all. Testcontainers resolves a + service's host *through* one of its published ports, and asked for a service + without naming one it demands that the service publish exactly one -- so on + a stack like flagd's, which publishes three on one service, the no-port form + raises ``NoSuchPortExposed`` and the message blames the port rather than the + call. Any of the service's ports answers the question, so the first declared + one is used. + """ + + @property + def host(self) -> str: + """The host the backend service is reachable on. + + Not necessarily ``localhost``: with a remote Docker daemon, Docker + Desktop on some platforms, or a rootless setup it can be an arbitrary + address. Always use this rather than hard-coding a host. + """ + return self.host_of(self.backend_service) + + def host_of(self, service: str) -> str: + """The host a named service is reachable on. + + :raises ControlApiError: if the declaration names no port for that + service, so there is nothing to resolve a host through -- and + nothing for a provider to connect to either. + """ + probe = next(iter(self.service_ports.get(service, ())), None) + if probe is None: + known = ", ".join(sorted(self.service_ports)) or "(none)" + msg = ( + f"no port is declared for service {service!r}, so its host cannot " + f"be resolved: a host is resolved through one of a service's " + f"published ports. Declared services are {known}; add it to " + f"ComposeBackend.additional_ports" + ) + raise ControlApiError(msg) + host = self.stack.get_service_host(service, probe) + return host or "localhost" + + def port(self, internal_port: int, *, service: str | None = None) -> int: + """The mapped host port for a container-internal port. + + :param internal_port: the container-internal port, as declared in + :attr:`ComposeBackend.backend_ports` or + :attr:`ComposeBackend.additional_ports`. + :param service: the Compose service, defaulting to + :attr:`backend_service`. Use it for a stack with more than one + service -- a proxy, an edge service, a sidecar. + :raises ControlApiError: if that port is not published by the stack, + which means the Compose file does not list it under ``ports:``. + """ + name = service or self.backend_service + mapped = self.stack.get_service_port(name, internal_port) + if mapped is None: + msg = ( + f"the Compose stack publishes no host port for {internal_port} on " + f"service {name!r}. A port is only published if the Compose file " + f"lists it under that service's `ports:` -- unpinned, as a bare " + f"container port, so Docker maps it dynamically" + ) + raise ControlApiError(msg) + return int(mapped) + + +@dataclass(frozen=True) +class ComposeBackend: + """What an adopter declares about the stack under test. + + Everything except :attr:`compose_file` and :attr:`backend_ports` has a + default, and the defaults are the same in every language's TCK. + """ + + compose_file: str | Path + """Path to the Compose file describing the stack. + + Resolved relative to the package directory -- the directory ``pytest`` was + invoked from for a normal ``poe test``, which is the same directory the + package's ``pyproject.toml`` sits in. An absolute path is used as given. + + The stack must not pin host ports. Docker assigns them dynamically and this + module discovers them after startup; a pinned host port makes the suite + unrunnable in parallel and collides with whatever the developer already has + listening. + """ + + backend_ports: Sequence[int] + """Container-internal ports on :attr:`backend_service` that the *provider* + connects to. + + :attr:`control_port` is handled automatically and must not be listed here. + + Declaring them is what lets the harness fail with "the Compose file does not + publish 8013" at startup, rather than with a provider that cannot connect + three scenarios later. + """ + + backend_service: str = DEFAULT_BACKEND_SERVICE + """The Compose service hosting both the control API and the backend.""" + + control_port: int = DEFAULT_CONTROL_PORT + """The container-internal port of the control API.""" + + additional_ports: Mapping[str, Sequence[int]] = field(default_factory=dict) + """Extra service -> ports, for a stack with more than one service. + + Resolved through the endpoint by service name:: + + endpoint.port(9212, service="proxy") + """ + + configuration: str = DEFAULT_CONFIGURATION + """The configuration name passed to ``POST /start``. + + ``default`` is the only name every backend must support, and the one that + serves the canonical flag set the feature files assume. + """ + + startup_timeout: float = DEFAULT_STARTUP_TIMEOUT + """Seconds to wait for the stack and its control API to become reachable.""" + + def __post_init__(self) -> None: + # Normalised before anything is validated, so a declaration written as + # any other iterable is checked as the tuple it becomes rather than + # consumed by the checking. + object.__setattr__(self, "backend_ports", tuple(self.backend_ports)) + object.__setattr__( + self, + "additional_ports", + {service: tuple(ports) for service, ports in self.additional_ports.items()}, + ) + + problems: list[str] = [] + + if not str(self.compose_file): + problems.append( + "compose_file is required: it is the path to the Compose file " + "describing the stack under test" + ) + if not self.backend_ports: + problems.append( + "backend_ports is required and must not be empty: it is the " + "container-internal ports the provider connects to, so the harness can " + "check the Compose file publishes them before the first scenario. The " + f"control port ({self.control_port}) is handled automatically and does " + "not belong here" + ) + if self.control_port in self.backend_ports: + problems.append( + f"backend_ports lists the control port {self.control_port}: it is " + "exposed automatically, and a provider that connects to the control " + "API is not exercising the contract this suite tests. Remove it, or " + "set control_port if the control API is somewhere else" + ) + if not self.backend_service: + problems.append( + "backend_service must name a Compose service: it is the service " + "hosting both the control API and the backend" + ) + if self.startup_timeout <= 0: + problems.append( + f"startup_timeout must be positive, not {self.startup_timeout!r}" + ) + + if problems: + joined = "\n - ".join(problems) + msg = f"invalid ComposeBackend:\n - {joined}" + raise ValueError(msg) + + def resolved_compose_file(self, root: Path | None = None) -> Path: + """The Compose file as an absolute path, resolved against ``root``. + + ``root`` defaults to the current working directory, which for a normal + ``poe test`` is the package directory. + """ + candidate = Path(self.compose_file) + if not candidate.is_absolute(): + candidate = (root or Path.cwd()) / candidate + return candidate + + @property + def exposed_ports(self) -> dict[str, tuple[int, ...]]: + """Every service and container-internal port the stack must publish. + + The control port first, because a stack that publishes nothing else + still has to answer control calls, then the backend ports, then whatever + :attr:`additional_ports` adds. + """ + ports: dict[str, tuple[int, ...]] = { + self.backend_service: (self.control_port, *self.backend_ports), + } + for service, extra in self.additional_ports.items(): + merged = (*ports.get(service, ()), *extra) + # dict.fromkeys rather than a set: order is what makes the startup + # failure message read in the order the adopter wrote the ports. + ports[service] = tuple(dict.fromkeys(merged)) + return ports + + +@dataclass(frozen=True) +class RunningBackend: + """The started stack, as the ``tck_backend`` fixture yields it. + + Two things, because two things are all an adoption needs: the control to + hand to :attr:`TckConfig.control`, and the endpoint to build providers from. + """ + + control: HttpControl + """The control API client, already awaited ready. + + One instance per stack, and it must stay that way where two suites drive the + same backend: it remembers whether a disconnect has left the backend down, + so the next scenario is prepared with ``/start`` rather than ``/reset``, and + two instances would each hold half of that knowledge. + """ + + endpoint: BackendEndpoint + """Host and mapped ports of the running stack, for the provider factory.""" + + backend: ComposeBackend + """The declaration this stack was started from.""" + + +def run_compose_backend( + backend: ComposeBackend, *, root: Path | None = None +) -> Iterator[RunningBackend]: + """Start the declared stack, yield it, and tear it down. + + A generator, so the ``tck_backend`` fixture is ``yield from`` over this and + an adopter who wants a fixture of their own naming or scoping can be too:: + + @pytest.fixture(scope="session") + def tck_backend() -> Iterator[RunningBackend]: + yield from run_compose_backend(ComposeBackend(...)) + + Startup is: bring the stack up and wait for its containers, wait for every + declared port to accept a connection, then wait for the control API itself + to accept commands. The last of those is a real readiness check rather than + a pause -- see :meth:`HttpControl.await_ready`. + """ + compose_file = backend.resolved_compose_file(root) + if not compose_file.is_file(): + msg = ( + f"Compose file not found: {compose_file}. " + f"ComposeBackend.compose_file is resolved relative to the package " + f"directory, which is where pytest runs from" + ) + raise FileNotFoundError(msg) + + stack = _docker_compose(compose_file) + stack.start() + try: + endpoint = BackendEndpoint( + stack=typing.cast("ComposeStack", stack), + backend_service=backend.backend_service, + service_ports=backend.exposed_ports, + ) + _await_ports(backend, endpoint) + + control = HttpControl( + f"http://{endpoint.host}:{endpoint.port(backend.control_port)}", + configuration=backend.configuration, + ) + control.await_ready(backend.startup_timeout) + + yield RunningBackend(control=control, endpoint=endpoint, backend=backend) + finally: + stack.stop() + + +def _docker_compose(compose_file: Path) -> typing.Any: + """Build a ``DockerCompose`` for one Compose file. + + Imported here rather than at module scope so that importing this module -- + which the package's ``__init__`` does -- costs nothing and, more to the + point, does not require ``testcontainers`` to be installed. An in-memory + adopter has no use for container tooling and should not have to install it. + """ + try: + # PLC0415: deliberately not at module scope. That is the whole point of + # this function -- see the docstring. + from testcontainers.compose import DockerCompose # noqa: PLC0415 + except ImportError as error: # pragma: no cover - depends on the environment + msg = ( + "the Compose harness needs testcontainers, which is the `compose` " + "extra of this package: `pip install 'openfeature-tck[compose]'`. It is " + "optional because a provider with no backend runs the whole suite " + "without a container -- see the in-process control" + ) + raise ImportError(msg) from error + + return DockerCompose( + context=str(compose_file.parent), + compose_file_name=compose_file.name, + # `docker compose up --wait`, so start() returns once the containers are + # up rather than once the command has been issued. + wait=True, + ) + + +def _await_ports(backend: ComposeBackend, endpoint: BackendEndpoint) -> None: + """Wait until every declared port accepts a TCP connection. + + Java's harness gets this from a Testcontainers listening-port wait strategy + per exposed service port; ``docker compose up --wait`` only promises the + container is up, which for a service with no healthcheck it is well before + anything is listening. Same guarantee, established the same way, so the two + languages fail at the same point rather than one of them failing later and + somewhere less obvious. + """ + deadline = time.monotonic() + backend.startup_timeout + for service, ports in backend.exposed_ports.items(): + host = endpoint.host_of(service) + for internal in ports: + mapped = endpoint.port(internal, service=service) + _await_listening(host, mapped, service, internal, deadline) + + +def _await_listening( + host: str, port: int, service: str, internal: int, deadline: float +) -> None: + last: OSError | None = None + while True: + try: + with contextlib.closing( + socket.create_connection((host, port), timeout=_PORT_PROBE_TIMEOUT) + ): + return + except OSError as error: + last = error + if time.monotonic() >= deadline: + msg = ( + f"nothing is listening on {host}:{port} -- the host port Compose " + f"mapped for container port {internal} of service {service!r} -- " + f"within the startup timeout: {last}. Either the service does not " + f"listen on {internal}, or the stack needs a longer " + f"ComposeBackend.startup_timeout" + ) + raise ControlApiError(msg) + time.sleep(_PORT_POLL_INTERVAL) diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py index 5e787b5e7..ab1a9ec1a 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py @@ -3,6 +3,7 @@ from __future__ import annotations import threading +import time import typing import urllib.error import urllib.parse @@ -10,7 +11,12 @@ from .control import BackendControl, ConnectionControl -__all__ = ["DEFAULT_CONFIGURATION", "ControlApiError", "HttpControl"] +__all__ = [ + "DEFAULT_CONFIGURATION", + "DEFAULT_STARTUP_TIMEOUT", + "ControlApiError", + "HttpControl", +] DEFAULT_CONFIGURATION = "default" """The configuration name every backend under test must support. @@ -25,11 +31,24 @@ than this is a wedged backend rather than a slow one. """ +DEFAULT_STARTUP_TIMEOUT = 60.0 +"""Seconds to wait for a stack and its control API to become reachable. + +The same default every language's TCK uses, so an adopter porting an adoption +between two of them does not find one of them more patient than the other. +""" + _NOT_IMPLEMENTED = frozenset({404, 501}) """How a backend that does not implement ``/reset`` answers it, per the OpenAPI document.""" _SUPPORTED_SCHEMES = frozenset({"http", "https"}) +_READY_PROBE_TIMEOUT = 5.0 +"""Seconds bounding a single readiness probe, so one wedged probe is not the whole wait.""" + +_READY_POLL_INTERVAL = 0.2 +"""Seconds between readiness probes.""" + class ControlApiError(RuntimeError): """Raised when a control-API call fails or answers with an unexpected status. @@ -134,6 +153,70 @@ def control_api(self) -> str: def description(self) -> str: return f"the backend at {self._base_url}, driven over the control API" + @property + def base_url(self) -> str: + """The root of the control API this client drives.""" + return self._base_url + + def await_ready(self, timeout: float = DEFAULT_STARTUP_TIMEOUT) -> None: + """Block until the control API is ready to accept commands. + + A real readiness check against the control API itself rather than a + fixed pause, and the only wait in this class. ``GET /healthz`` is the + optional readiness path in ``control-api.yaml``; a backend that does not + implement it answers 404, which the document states *is* ready -- + readiness then rests on the control port accepting a connection, which + whatever started the stack has already established. A 503 is the control + API saying "not yet" and is retried. + + Called once, before the first command, by whatever brought the stack up. + There is deliberately no counterpart *after* a command: a pause there + would cover a window the control API is specified to close on its own, + and a suite that sleeps instead of holding the API to that promise stops + being able to detect when the promise breaks. + + :param timeout: seconds to keep probing before giving up. + :raises ControlApiError: if the control API is still not ready when the + timeout expires, quoting the last thing the probe saw. + """ + deadline = time.monotonic() + timeout + last = "no probe completed" + while True: + try: + status = self._probe_health() + except OSError as error: + last = f"not reachable: {error}" + else: + # 404 is "not implemented", which the document defines as ready. + if self._is_success(status) or status == 404: + return + last = f"answered HTTP {status}" + + if time.monotonic() >= deadline: + msg = ( + f"the control API at {self._base_url} was not ready within " + f"{timeout:g}s: GET /healthz {last}. The control API must be " + f"reachable before the first scenario and must stay reachable " + f"even while the backend is deliberately down" + ) + raise ControlApiError(msg) + time.sleep(_READY_POLL_INTERVAL) + + def _probe_health(self) -> int: + target = self._base_url + "/healthz" + # S310: __init__ rejects any base_url that is not http(s), and target is + # that validated base URL plus a literal path. + request = urllib.request.Request(target, method="GET") # noqa: S310 + try: + opened = urllib.request.urlopen(request, timeout=_READY_PROBE_TIMEOUT) # noqa: S310 + with opened as response: + response.read() + return int(response.status) + except urllib.error.HTTPError as error: + with error: + error.read() + return int(error.code) + def prepare_scenario(self) -> None: """Bring the backend to the state every scenario starts from. diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py index 18103e981..791111cef 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py @@ -21,6 +21,7 @@ from openfeature import api from .capability import Capability, capability_for_marker +from .compose import ComposeBackend, RunningBackend, run_compose_backend from .config import TckConfig from .state import TckState @@ -51,6 +52,37 @@ def pytest_configure(config: pytest.Config) -> None: ) +@pytest.fixture(scope="session") +def tck_backend(compose_backend: ComposeBackend) -> typing.Iterator[RunningBackend]: + """The Compose stack under test, started once for the whole session. + + Depends on an adopter-supplied ``compose_backend`` fixture returning a + :class:`~.compose.ComposeBackend`, and yields the started stack's control + and endpoint. An adoption then reads:: + + @pytest.fixture(scope="session") + def compose_backend() -> ComposeBackend: + return ComposeBackend( + compose_file="tests/tck/docker-compose.yaml", + backend_ports=[8013], + ) + + @pytest.fixture(scope="session") + def tck_config(tck_backend: RunningBackend) -> TckConfig: + ... + + Session-scoped rather than module-scoped on purpose: two suites that drive + the same backend -- flagd's two resolvers, say -- must share one stack *and* + one control, because the control remembers whether a disconnect left the + backend down and two instances would each hold half of that knowledge. + + Lazy, like every fixture: a provider with no backend never requests it, + never defines ``compose_backend``, and never needs Docker or the ``compose`` + extra installed. See :mod:`~.compose`. + """ + yield from run_compose_backend(compose_backend) + + @pytest.fixture def tck_state(tck_config: TckConfig) -> typing.Iterator[TckState]: """Per-scenario state, carried between step definitions.""" diff --git a/tools/openfeature-tck/tests/test_compose.py b/tools/openfeature-tck/tests/test_compose.py new file mode 100644 index 000000000..df10e9b71 --- /dev/null +++ b/tools/openfeature-tck/tests/test_compose.py @@ -0,0 +1,352 @@ +"""What an adopter declares about a container stack, and what the harness owes them. + +The stack lifecycle itself needs Docker and is proved by the flagd adoption, +which runs the whole canonical suite against a real testbed through this +harness. What is checked here is everything that can be wrong *without* a +container, and all of it was adopter code until now: the declaration's defaults, +the refusals that turn a mistake into a message instead of a provider that +cannot connect three scenarios later, and the port resolution the provider +factory is handed. + +``ComposeStack`` is a protocol rather than ``DockerCompose`` itself, which is +what makes that possible: a stub satisfying two methods stands in for Docker, so +the resolution logic is examined rather than assumed. +""" + +from __future__ import annotations + +import dataclasses +import typing +from pathlib import Path + +import pytest + +from openfeature.contrib.tools.tck import ( + DEFAULT_BACKEND_SERVICE, + DEFAULT_CONFIGURATION, + DEFAULT_CONTROL_PORT, + DEFAULT_STARTUP_TIMEOUT, + BackendEndpoint, + ComposeBackend, + ControlApiError, + run_compose_backend, +) + +BACKEND_PORT = 8013 +"""A container-internal port a provider connects to. flagd's RPC port, for realism.""" + + +@dataclasses.dataclass +class _StubStack: + """A Compose stack that publishes exactly what it is told to. + + Keyed by ``(service, container port) -> host port``, because that is the + whole of what the harness asks a stack for -- and getting the service half + wrong is how a multi-service stack resolves a port against the wrong + container. + """ + + published: dict[tuple[str, int], int] + host: str = "127.0.0.1" + + def get_service_host( + self, service_name: str | None = None, port: int | None = None + ) -> str | None: + # Testcontainers resolves a host *through* a published port, and asked + # without one it insists the service publish exactly one. Reproduced + # rather than waved through: a stub that answered anyway is why the + # no-port form reached a real three-port stack and raised there. + assert service_name is not None + if port is None: + ports = [key for key in self.published if key[0] == service_name] + assert len(ports) == 1, ( + f"service {service_name!r} publishes {len(ports)} ports, so the " + f"host cannot be resolved without naming one of them" + ) + return self.host + + def get_service_port( + self, service_name: str | None = None, port: int | None = None + ) -> int | None: + assert service_name is not None + assert port is not None + return self.published.get((service_name, port)) + + +def _endpoint( + published: dict[tuple[str, int], int], + service: str = DEFAULT_BACKEND_SERVICE, +) -> BackendEndpoint: + """An endpoint over a stub stack, told which ports each service publishes. + + Which is what the real one is told, from the declaration: the host is + resolved through one of them. + """ + ports: dict[str, list[int]] = {} + for name, internal in published: + ports.setdefault(name, []).append(internal) + return BackendEndpoint( + stack=typing.cast("typing.Any", _StubStack(published=published)), + backend_service=service, + service_ports=ports, + ) + + +# -- the declaration --------------------------------------------------------- + + +def test_the_defaults_are_the_ones_every_language_fixes() -> None: + """Pinned as literals, because the Compose file is what an adoption shares. + + A provider shipped in two languages writes one Compose file and two + declarations against it, so a default that differs between two TCKs makes + the file wrong in one of them -- and the symptom is a stack that starts and + a control API nothing can reach. Every other assertion in this file uses the + constants and would stay green through a change to them. + """ + backend = ComposeBackend(compose_file="docker-compose.yaml", backend_ports=[8013]) + + assert backend.backend_service == "backend" + assert backend.control_port == 8080 + assert backend.additional_ports == {} + assert backend.configuration == "default" + assert backend.startup_timeout == 60.0 + + assert DEFAULT_BACKEND_SERVICE == "backend" + assert DEFAULT_CONTROL_PORT == 8080 + assert DEFAULT_CONFIGURATION == "default" + assert DEFAULT_STARTUP_TIMEOUT == 60.0 + + +def test_backend_ports_is_required_and_says_the_control_port_is_not_one() -> None: + """The one refusal an adopter is most likely to meet, so it has to teach. + + An empty ``backend_ports`` is a declaration that says nothing about which + ports the provider needs, which turns a startup check into nothing and the + first scenario into an unexplained connection failure. + """ + with pytest.raises(ValueError, match="backend_ports is required") as raised: + ComposeBackend(compose_file="docker-compose.yaml", backend_ports=[]) + + assert "control port (8080) is handled automatically" in str(raised.value) + + +def test_the_control_port_may_not_be_declared_as_a_backend_port() -> None: + """Listing it is a sign of a misunderstanding rather than a duplicate entry. + + The control API is the harness's, not the provider's. A provider pointed at + it is not exercising the contract this suite tests, and a declaration that + lists it reads as though it might be. + """ + with pytest.raises(ValueError, match="lists the control port 8080"): + ComposeBackend( + compose_file="docker-compose.yaml", + backend_ports=[BACKEND_PORT, DEFAULT_CONTROL_PORT], + ) + + +def test_a_relocated_control_port_may_be_a_backend_port_elsewhere() -> None: + """The check is against the declared control port, not against 8080. + + A stack serving its control API somewhere else is entitled to have 8080 be + an ordinary backend port. + """ + backend = ComposeBackend( + compose_file="docker-compose.yaml", + backend_ports=[DEFAULT_CONTROL_PORT], + control_port=9090, + ) + assert backend.exposed_ports == {"backend": (9090, DEFAULT_CONTROL_PORT)} + + +def test_a_declaration_reports_every_problem_it_has_at_once() -> None: + """One traceback naming everything wrong, not one per round trip.""" + with pytest.raises(ValueError) as raised: + ComposeBackend( + compose_file="docker-compose.yaml", + backend_ports=[], + backend_service="", + startup_timeout=0, + ) + + message = str(raised.value) + assert "backend_ports is required" in message + assert "backend_service must name a Compose service" in message + assert "startup_timeout must be positive" in message + + +def test_the_declaration_is_normalised_to_tuples() -> None: + """So a declaration written as a list is stored as what it is checked as. + + Normalised before validation rather than after, which is the order that + matters: the control-port check reads ``backend_ports``, and reading it + first would consume a one-shot iterable before it reached the field. + """ + backend = ComposeBackend( + compose_file="docker-compose.yaml", + backend_ports=[8013, 8015], + additional_ports={"proxy": [9212]}, + ) + assert backend.backend_ports == (8013, 8015) + assert backend.additional_ports == {"proxy": (9212,)} + + +def test_the_control_port_is_exposed_ahead_of_everything_an_adopter_named() -> None: + """It is the harness's own port and is never declared, so it is added here. + + First, because a stack that publishes nothing else still has to answer + control calls, and the startup failure should say so before it says anything + about a provider port. + """ + backend = ComposeBackend( + compose_file="docker-compose.yaml", + backend_ports=[8013, 8015], + additional_ports={"proxy": [9212]}, + ) + assert backend.exposed_ports == { + "backend": (DEFAULT_CONTROL_PORT, 8013, 8015), + "proxy": (9212,), + } + + +def test_additional_ports_on_the_backend_service_join_rather_than_replace() -> None: + """A stack may name the backend service again without losing the control port.""" + backend = ComposeBackend( + compose_file="docker-compose.yaml", + backend_ports=[8013], + additional_ports={DEFAULT_BACKEND_SERVICE: [8015, 8013]}, + ) + assert backend.exposed_ports == {"backend": (DEFAULT_CONTROL_PORT, 8013, 8015)} + + +# -- resolving the compose file ---------------------------------------------- + + +def test_a_relative_compose_path_resolves_against_the_package_directory( + tmp_path: Path, +) -> None: + """Which is where pytest runs from, so ``tests/tck/docker-compose.yaml`` works.""" + backend = ComposeBackend( + compose_file="tests/tck/docker-compose.yaml", backend_ports=[BACKEND_PORT] + ) + assert ( + backend.resolved_compose_file(tmp_path) + == tmp_path / "tests/tck/docker-compose.yaml" + ) + + +def test_an_absolute_compose_path_is_used_as_given(tmp_path: Path) -> None: + absolute = tmp_path / "elsewhere" / "docker-compose.yaml" + backend = ComposeBackend(compose_file=absolute, backend_ports=[BACKEND_PORT]) + assert backend.resolved_compose_file(tmp_path / "ignored") == absolute + + +def test_a_missing_compose_file_fails_before_anything_is_started( + tmp_path: Path, +) -> None: + """And says where it looked, because the answer is usually "not where I meant". + + Raised from the generator before Docker is touched, so an adopter who + mistyped the path does not wait for a stack to come up first -- and never + needs Docker to find out. + """ + backend = ComposeBackend( + compose_file="tests/tck/docker-compose.yaml", backend_ports=[BACKEND_PORT] + ) + with pytest.raises(FileNotFoundError) as raised: + next(run_compose_backend(backend, root=tmp_path)) + + message = str(raised.value) + assert "docker-compose.yaml" in message + assert "resolved relative to the package directory" in message + + +# -- the endpoint the provider factory is handed ----------------------------- + + +def test_a_mapped_port_is_looked_up_by_container_port() -> None: + """Which is what an adopter knows: 8013 is in their Compose file, 32769 is not.""" + endpoint = _endpoint({(DEFAULT_BACKEND_SERVICE, BACKEND_PORT): 32769}) + assert endpoint.port(BACKEND_PORT) == 32769 + + +def test_a_port_may_be_qualified_by_service_for_a_multi_service_stack() -> None: + """Two services may publish the same container port, and usually do. + + Resolving one against the other is silent: the provider connects to + something that answers, and the scenario fails on whatever it answers with. + """ + endpoint = _endpoint( + { + (DEFAULT_BACKEND_SERVICE, BACKEND_PORT): 32769, + ("proxy", BACKEND_PORT): 32770, + } + ) + assert endpoint.port(BACKEND_PORT) == 32769 + assert endpoint.port(BACKEND_PORT, service="proxy") == 32770 + + +def test_the_host_is_the_stacks_own_rather_than_localhost() -> None: + """A remote daemon, Docker Desktop or a rootless setup can serve any address.""" + endpoint = BackendEndpoint( + stack=typing.cast( + "typing.Any", + _StubStack( + published={(DEFAULT_BACKEND_SERVICE, BACKEND_PORT): 32769}, + host="192.168.64.2", + ), + ), + service_ports={DEFAULT_BACKEND_SERVICE: [BACKEND_PORT]}, + ) + assert endpoint.host == "192.168.64.2" + + +def test_the_host_of_a_service_publishing_several_ports_still_resolves() -> None: + """flagd's stack publishes three on one service, which is the common shape. + + Resolved through one of the declared ports, because testcontainers resolves + a host *through* a published port and, asked without one, requires the + service to publish exactly one. The no-port form therefore worked on every + single-port stack and raised ``NoSuchPortExposed`` on the first real one -- + with a message about the port rather than about the call. + """ + endpoint = _endpoint( + { + (DEFAULT_BACKEND_SERVICE, DEFAULT_CONTROL_PORT): 32768, + (DEFAULT_BACKEND_SERVICE, BACKEND_PORT): 32769, + (DEFAULT_BACKEND_SERVICE, 8015): 32770, + } + ) + assert endpoint.host == "127.0.0.1" + assert endpoint.port(8015) == 32770 + + +def test_an_undeclared_service_says_so_rather_than_failing_inside_docker() -> None: + """A service the declaration does not name has nothing published for it. + + So there is nothing to resolve a host through and nothing for a provider to + connect to. Answered here, naming the services that *were* declared, rather + than passed down to testcontainers to answer as a port problem. + """ + endpoint = _endpoint({(DEFAULT_BACKEND_SERVICE, BACKEND_PORT): 32769}) + with pytest.raises(ControlApiError) as raised: + endpoint.host_of("proxy") + + message = str(raised.value) + assert "no port is declared for service 'proxy'" in message + assert "additional_ports" in message + + +def test_an_unpublished_port_says_the_compose_file_has_to_list_it() -> None: + """The error an adopter actually hits, and the fix is in the Compose file. + + Testcontainers' own answer here is a ``NoSuchPortExposed`` naming the port + and nothing else, which reads as though the harness were at fault. + """ + endpoint = _endpoint({(DEFAULT_BACKEND_SERVICE, DEFAULT_CONTROL_PORT): 32768}) + with pytest.raises(ControlApiError) as raised: + endpoint.port(BACKEND_PORT) + + message = str(raised.value) + assert f"no host port for {BACKEND_PORT}" in message + assert "`ports:`" in message diff --git a/tools/openfeature-tck/tests/test_http_control.py b/tools/openfeature-tck/tests/test_http_control.py index 09b9d03f1..fc72e4248 100644 --- a/tools/openfeature-tck/tests/test_http_control.py +++ b/tools/openfeature-tck/tests/test_http_control.py @@ -28,18 +28,36 @@ class _StubControlApi: """A control API that records every request and answers a scripted status.""" - def __init__(self, statuses: dict[str, int] | None = None) -> None: + def __init__( + self, + statuses: dict[str, int] | None = None, + sequences: dict[str, list[int]] | None = None, + ) -> None: self.requests: list[tuple[str, str, str]] = [] """(method, path, query) of every request, in order.""" self.statuses = statuses or {} + self.sequences = sequences or {} + """Per-path statuses consumed one per request, for "not yet, then yes".""" + stub = self class Handler(BaseHTTPRequestHandler): def do_POST(self) -> None: + self._answer("POST") + + def do_GET(self) -> None: + self._answer("GET") + + def _answer(self, method: str) -> None: path, _, query = self.path.partition("?") - stub.requests.append(("POST", path, query)) + stub.requests.append((method, path, query)) status = stub.statuses.get(path, 200) + # Scripted as a list to answer differently on each call, which + # is how "not ready, then ready" is expressed. + sequence = stub.sequences.get(path) + if sequence: + status = sequence.pop(0) body = b'{"status":"stub"}' self.send_response(status) self.send_header("Content-Type", "application/json") @@ -232,3 +250,78 @@ def test_the_control_reports_which_api_it_drives_the_backend_through() -> None: """ with _StubControlApi() as stub: assert HttpControl(stub.base_url).control_api == "http" + + +# -- waiting for the control API --------------------------------------------- + + +def test_await_ready_returns_as_soon_as_healthz_answers( + stub: _StubControlApi, +) -> None: + """One probe against the control API itself, not a pause of a fixed length. + + The readiness check is what replaced Java's post-command settle. It probes + the thing whose readiness is in question, so a control API that is slow to + come up is waited for and one that never does is reported. + """ + control = HttpControl(stub.base_url) + + control.await_ready(timeout=5.0) + + assert stub.requests == [("GET", "/healthz", "")] + + +def test_await_ready_treats_an_unimplemented_healthz_as_ready() -> None: + """404 is "not implemented", which ``control-api.yaml`` defines as ready. + + Readiness then rests on the control port accepting a connection, which the + Compose harness has already established before it gets here. The reference + backend -- flagd-testbed's launchpad -- serves no ``/healthz`` at all, so + this is the normal path today rather than an edge case. + """ + with _StubControlApi({"/healthz": 404}) as stub: + HttpControl(stub.base_url).await_ready(timeout=5.0) + assert stub.paths == ["/healthz"] + + +def test_await_ready_keeps_probing_while_the_control_api_says_not_yet() -> None: + """503 is the control API saying "not ready", so it is retried, not accepted.""" + with _StubControlApi(sequences={"/healthz": [503, 503, 200]}) as stub: + HttpControl(stub.base_url).await_ready(timeout=10.0) + assert stub.paths == ["/healthz", "/healthz", "/healthz"] + + +def test_await_ready_gives_up_with_what_the_last_probe_saw() -> None: + """A stack that never becomes ready has to say what it was answering. + + "did not become ready" on its own sends an adopter to the wrong place: a + connection refused is a stack that is not up, a 503 is one that is up and + not finished. + """ + with _StubControlApi({"/healthz": 503}) as stub: + control = HttpControl(stub.base_url) + with pytest.raises(ControlApiError) as raised: + control.await_ready(timeout=0.3) + + message = str(raised.value) + assert "was not ready within" in message + assert "HTTP 503" in message + + +def test_await_ready_reports_a_control_api_that_is_not_there_at_all() -> None: + """Which is a stack that did not start, and reads differently from a 503.""" + with _StubControlApi() as stub: + base_url = stub.base_url + # The server is closed, so nothing is listening on that port any more. + + control = HttpControl(base_url) + with pytest.raises(ControlApiError) as raised: + control.await_ready(timeout=0.3) + + assert "not reachable" in str(raised.value) + + +def test_base_url_is_reportable_without_rebuilding_it() -> None: + """Whatever brought the stack up logs where the control API ended up.""" + with _StubControlApi() as stub: + assert HttpControl(stub.base_url + "/").base_url == stub.base_url diff --git a/uv.lock b/uv.lock index 4c723c6fe..ea025adb2 100644 --- a/uv.lock +++ b/uv.lock @@ -2095,11 +2095,17 @@ dependencies = [ { name = "pytest-bdd" }, ] +[package.optional-dependencies] +compose = [ + { name = "testcontainers" }, +] + [package.dev-dependencies] dev = [ { name = "coverage", extra = ["toml"] }, { name = "mypy" }, { name = "poethepoet" }, + { name = "testcontainers" }, ] [package.metadata] @@ -2107,13 +2113,16 @@ requires-dist = [ { name = "openfeature-sdk", specifier = ">=0.10.0" }, { name = "pytest", specifier = ">=8.4.0" }, { name = "pytest-bdd", specifier = ">=8.1.0,<9.0.0" }, + { name = "testcontainers", marker = "extra == 'compose'", specifier = ">=4.12.0,<5.0.0" }, ] +provides-extras = ["compose"] [package.metadata.requires-dev] dev = [ { name = "coverage", extras = ["toml"], specifier = ">=7.10.0,<8.0.0" }, { name = "mypy", specifier = ">=1.18.0,<2.0.0" }, { name = "poethepoet", specifier = ">=0.37.0" }, + { name = "testcontainers", specifier = ">=4.12.0,<5.0.0" }, ] [[package]] From fb65f80020d122e9fcdd0575aa9a601c6a4e06f8 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 16:24:05 +0200 Subject: [PATCH 27/46] chore(tck): re-pin the spec to 93eb1a58 The pin was parked at 009afe06 while the specification's changes were prose-only. control-api.yaml has now changed, and it is one of the three normative artifacts this package consumes, so the pin is due. What moved, in the assets: - every state-changing endpoint -- /start, /change, /reset -- must now not return until the new state is actually being served. /start already said so; the other two did not; - for /change the promise is spelled out as being about the *backend*, with the provider's own detection latency explicitly the business of the event timeout instead; - /restart is demoted to [OPTIONAL], because its own description falsely claimed the TCK used it for the disconnect/reconnect scenarios. The vendored copies under src/.../tck are gitignored and rebuilt by hatch_build_sync.py, so the pin is the whole of the change here. The feature files are untouched by this revision apart from a comment on @disabled-flags; no scenario and no canonical flag changed, so the suite's tally is unaffected. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/spec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/openfeature-tck/spec b/tools/openfeature-tck/spec index 009afe061..93eb1a58d 160000 --- a/tools/openfeature-tck/spec +++ b/tools/openfeature-tck/spec @@ -1 +1 @@ -Subproject commit 009afe0617947121dcbebe4b66e0cc0c5cc4ada8 +Subproject commit 93eb1a58d2d2ec015acb298c914c5822e7a38dd2 From 7fdd29690dfd723d334bbe7717265b9b4a594227 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 16:24:48 +0200 Subject: [PATCH 28/46] feat(tck)!: a control must say which path it drove the backend through control_api graduates from a comment block describing an optional duck-typed property to a required member of the BackendControl protocol, typed ControlApi -- Literal["http", "in-process"] -- with no default and no inference from the control's concrete type. The comment it replaces argued for omission on two grounds and both are void. "Adding one would make every existing control incomplete": nothing is published, and after the compose harness an adopter with a real backend writes no control at all -- the HTTP one comes with the harness. The only person who writes a control by hand is the one writing a custom one, which is precisely the case where the value cannot be inferred. "There is nothing useful the TCK can do with a control that has not said": correct, and it is the argument for requiring the control to say rather than for omitting the field. It is the one fact that decides what everything else in a report is worth. The same scenarios passing over the normative control API and passing through in-process manipulation of a provider that does have a backend are not the same claim, and this is the only field that separates them. Every run is one or the other, so an absent value is not "no claim made" but an unfalsifiable one. Closed rather than a bare str, so "HTTP" or "grpc" is a type error here instead of a conformance report that fails schema validation somewhere with nothing local to point at. ControlApi is exported for a custom control to annotate with. BackendControl is runtime-checkable, so a control that will not say is refused at the seam as well as by the type checker, which is what reaches an adopter whose test module is untyped. Appendix F states the same rule normatively at spec@93eb1a58: the control states the path, the harness must not guess, and an omitted value is not neutral. Signed-off-by: Simon Schrottner --- .../openfeature/contrib/tools/tck/__init__.py | 2 + .../openfeature/contrib/tools/tck/control.py | 47 ++++++++++++----- .../contrib/tools/tck/inprocess.py | 3 +- .../openfeature-tck/tests/test_declaration.py | 51 +++++++++++++++---- .../tests/test_in_memory_conformance.py | 6 +++ .../tests/test_lifecycle_steps.py | 5 ++ 6 files changed, 89 insertions(+), 25 deletions(-) diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py index c20926ba0..7572ae67a 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py @@ -94,6 +94,7 @@ def tck_config(): from .control import ( BackendControl, ConnectionControl, + ControlApi, UnsupportedControlError, ) from .extensions import ( @@ -130,6 +131,7 @@ def tck_config(): "Capability", "ComposeBackend", "ConnectionControl", + "ControlApi", "ControlApiError", "ControllableInMemoryProvider", "HttpControl", diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/control.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/control.py index e92dd18d4..56f3a34ce 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/control.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/control.py @@ -7,10 +7,20 @@ __all__ = [ "BackendControl", "ConnectionControl", + "ControlApi", "UnsupportedControlError", "unsupported_control", ] +ControlApi = typing.Literal["http", "in-process"] +"""Which of the two control paths a run used, closed to the two the schema allows. + +Named so that a custom control can annotate its own property with it and have +the type checker refuse a third value -- ``"HTTP"``, ``"grpc"``, a typo -- before +it becomes a conformance report that fails validation with nothing to point at +locally. +""" + class UnsupportedControlError(RuntimeError): """Raised when a backend cannot perform a control operation. @@ -73,19 +83,30 @@ def change_flag(self) -> None: def description(self) -> str: """A short description of what is being controlled, for messages a human reads.""" - # OPTIONAL: ``control_api`` - # - # A control may also offer a ``control_api`` property returning ``"http"`` - # for the normative HTTP control API, or ``"in-process"`` for the narrow - # allowance made for providers with no backend. The conformance report - # records it, so that a claim of in-process control by a provider that does - # have a backend can be treated with the suspicion it deserves. - # - # It is deliberately not a member of this protocol. Adding one would make - # every existing control incomplete for the sake of one string, and there is - # nothing useful the TCK can do with a control that has not said: it cannot - # tell from the outside whether a control spoke HTTP or reached into the - # process, so the field is simply omitted. See ``report.control_api_of``. + @property + def control_api(self) -> ControlApi: + """Which path this control drove the backend through. + + ``"http"`` is the normative HTTP control API in ``control-api.yaml``. + ``"in-process"`` is the narrow allowance made for a provider with no + backend, where "the backend" is a data structure in this process -- see + :class:`InProcessControl`. + + **Required, and stated rather than inferred.** It is the one fact that + decides what everything else in a report is worth: the same scenarios + passing over the control API and passing through in-process manipulation + of a provider that *does* have a backend are not the same claim, and + this is the only field that separates them. Nothing outside a control can + tell the two apart -- a suite that guessed from the control's concrete + type would be right about the two controls in this package and silently + wrong about a custom one, which is exactly the case where the answer + matters. + + Nor would an absent value be neutral. Every run is one or the other, so + there is no third case an omitted value legitimately covers: it would + not be "no claim made" but an unfalsifiable one. A custom control states + it here and nothing downstream has to guess. + """ @typing.runtime_checkable diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/inprocess.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/inprocess.py index 11586e0ff..f97d8b1a0 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/inprocess.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/inprocess.py @@ -4,6 +4,7 @@ from openfeature.provider import FeatureProvider +from .control import ControlApi from .provider import ( CHANGING_FLAG_KEY, ControllableInMemoryProvider, @@ -64,7 +65,7 @@ def description(self) -> str: return "in-process control of an in-memory provider" @property - def control_api(self) -> str: + def control_api(self) -> ControlApi: """Report how this backend was driven, for the conformance report. ``in-process`` is the narrow allowance for providers with no backend, diff --git a/tools/openfeature-tck/tests/test_declaration.py b/tools/openfeature-tck/tests/test_declaration.py index ba8d6a112..162e84959 100644 --- a/tools/openfeature-tck/tests/test_declaration.py +++ b/tools/openfeature-tck/tests/test_declaration.py @@ -31,6 +31,7 @@ RESERVED_CAPABILITIES, BackendControl, Capability, + ControlApi, InProcessControl, KnownDeviation, TckConfig, @@ -45,8 +46,8 @@ class _StubControl: """A control that says nothing it is not obliged to say. - Which includes ``control_api``: the property is documented as optional, and - a control leaving it out has to remain a ``BackendControl``. + Which no longer includes ``control_api``: it is a required member of + ``BackendControl``, so even a stub has to answer it. """ def prepare_scenario(self) -> None: ... @@ -57,6 +58,10 @@ def change_flag(self) -> None: ... def description(self) -> str: return "a stub" + @property + def control_api(self) -> ControlApi: + return "in-process" + def _config(**overrides: typing.Any) -> TckConfig: """A configuration that is valid but declares nothing in particular.""" @@ -334,17 +339,30 @@ def test_known_deviations_are_normalised_and_change_nothing_about_the_run() -> N # -- saying how the backend is driven ---------------------------------------- -def test_a_control_need_not_say_how_it_drives_the_backend() -> None: - """``control_api`` is documented as optional, and means it. +def test_a_control_that_does_not_say_is_not_a_backend_control() -> None: + """``control_api`` is required, and the protocol is where that is enforced. + + Nothing outside a control can tell whether it spoke the normative HTTP API + or reached into this process, which is the argument for making the control + say rather than for letting the field be absent: every run is one or the + other, so an omitted value is not "no claim made" but an unfalsifiable one. - Making it a member of ``BackendControl`` would make every existing control - incomplete for the sake of one string, and there is nothing the suite can do - with the answer: it cannot tell from the outside whether a control spoke - HTTP or reached into the process. + ``BackendControl`` is runtime-checkable, so this is checked at the seam as + well as by the type checker -- which matters for an adopter who writes a + custom control in an untyped test module. """ - quiet = _StubControl() - assert isinstance(quiet, BackendControl) - assert not hasattr(quiet, "control_api") + + class _Quiet: + def prepare_scenario(self) -> None: ... + + def change_flag(self) -> None: ... + + @property + def description(self) -> str: + return "a control that will not say" + + assert not isinstance(_Quiet(), BackendControl) + assert isinstance(_StubControl(), BackendControl) def test_in_process_control_says_it_is_in_process() -> None: @@ -356,3 +374,14 @@ def test_in_process_control_says_it_is_in_process() -> None: control = InProcessControl() assert isinstance(control, BackendControl) assert control.control_api == "in-process" + + +def test_the_control_api_type_is_closed_to_the_two_values_the_schema_allows() -> None: + """Closed, so a third value is a type error rather than an invalid report. + + ``ControlApi`` is exported for exactly this: a custom control annotates its + own property with it and the type checker refuses ``"HTTP"`` or ``"grpc"`` + before either becomes a report that fails schema validation with nothing to + point at locally. + """ + assert typing.get_args(ControlApi) == ("http", "in-process") diff --git a/tools/openfeature-tck/tests/test_in_memory_conformance.py b/tools/openfeature-tck/tests/test_in_memory_conformance.py index 1de4018ad..da63c6efd 100644 --- a/tools/openfeature-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-tck/tests/test_in_memory_conformance.py @@ -22,6 +22,7 @@ from openfeature.contrib.tools.tck import ( Capability, + ControlApi, TckConfig, canonical_flag_set, feature_paths, @@ -51,6 +52,11 @@ class PlainMemoryControl: def description(self) -> str: return "the Python SDK's InMemoryProvider, rebuilt per scenario" + @property + def control_api(self) -> ControlApi: + """In-process, and honestly so: there is no backend to speak HTTP to.""" + return "in-process" + def prepare_scenario(self) -> None: return None diff --git a/tools/openfeature-tck/tests/test_lifecycle_steps.py b/tools/openfeature-tck/tests/test_lifecycle_steps.py index 350b38d02..f556fedbd 100644 --- a/tools/openfeature-tck/tests/test_lifecycle_steps.py +++ b/tools/openfeature-tck/tests/test_lifecycle_steps.py @@ -26,6 +26,7 @@ from openfeature import api from openfeature.contrib.tools.tck import ( Capability, + ControlApi, TckConfig, TckState, canonical_flag_set, @@ -84,6 +85,10 @@ class _NoControl: def description(self) -> str: return "nothing" + @property + def control_api(self) -> ControlApi: + return "in-process" + def prepare_scenario(self) -> None: ... def change_flag(self) -> None: ... From c13f843b9ceb370212b22888c13c70b49fd30ebd Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 16:25:12 +0200 Subject: [PATCH 29/46] refactor(tck)!: the backend's configuration is backend_configuration ComposeBackend.configuration becomes backend_configuration, and HttpControl's keyword argument with it. The word was already taken. provider.configuration in the conformance report schema is "which configuration of the provider was tested, when a provider has more than one materially different mode" -- flagd RPC versus in-process -- and it is the field TckConfig.name feeds. That is the provider's mode, not the backend's config file, so one word for both made a report's configuration mean opposite things depending on which language's TCK produced it. Java distinguished them correctly from the start; Go, Python and JavaScript all took the word for the backend one. Settled as backendConfiguration in all four languages, spelled backend_configuration here. configuration keeps its schema meaning everywhere. DEFAULT_CONFIGURATION keeps its name: it names the default *value*, "default", which is the only configuration name every backend under test must support, and Go's DefaultConfiguration is spelled the same. Neither adoption passed the field -- both take the default -- so nothing outside this package had to change. Signed-off-by: Simon Schrottner --- .../src/openfeature/contrib/tools/tck/compose.py | 11 +++++++++-- tools/openfeature-tck/tests/test_compose.py | 2 +- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/compose.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/compose.py index 7e7f36397..09f0b3724 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/compose.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/compose.py @@ -256,11 +256,18 @@ class ComposeBackend: endpoint.port(9212, service="proxy") """ - configuration: str = DEFAULT_CONFIGURATION + backend_configuration: str = DEFAULT_CONFIGURATION """The configuration name passed to ``POST /start``. ``default`` is the only name every backend must support, and the one that serves the canonical flag set the feature files assume. + + Named for the *backend* because ``configuration`` on its own is already + taken, by the conformance report's ``provider.configuration`` -- which is + which materially different mode of the provider was tested, flagd's RPC + versus in-process, and which :attr:`TckConfig.name` feeds. The two are + unrelated and one word for both made a report's ``configuration`` mean + opposite things depending on which language's TCK produced it. """ startup_timeout: float = DEFAULT_STARTUP_TIMEOUT @@ -406,7 +413,7 @@ def tck_backend() -> Iterator[RunningBackend]: control = HttpControl( f"http://{endpoint.host}:{endpoint.port(backend.control_port)}", - configuration=backend.configuration, + backend_configuration=backend.backend_configuration, ) control.await_ready(backend.startup_timeout) diff --git a/tools/openfeature-tck/tests/test_compose.py b/tools/openfeature-tck/tests/test_compose.py index df10e9b71..f1b28b210 100644 --- a/tools/openfeature-tck/tests/test_compose.py +++ b/tools/openfeature-tck/tests/test_compose.py @@ -109,7 +109,7 @@ def test_the_defaults_are_the_ones_every_language_fixes() -> None: assert backend.backend_service == "backend" assert backend.control_port == 8080 assert backend.additional_ports == {} - assert backend.configuration == "default" + assert backend.backend_configuration == "default" assert backend.startup_timeout == 60.0 assert DEFAULT_BACKEND_SERVICE == "backend" From d62c722c5660eb2c6887908ab21a5737f1ec3f04 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 16:25:24 +0200 Subject: [PATCH 30/46] refactor(tck)!: remove the /restart binding, which nothing can reach HttpControl.restart is deleted. POST /restart is [OPTIONAL] in control-api.yaml as of spec@93eb1a58, because its own description falsely claimed the TCK used it for the disconnect/reconnect scenarios. It does not: the @stale scenario is written as an *unbounded* outage -- "the connection is lost", then "the connection is restored" -- which is /stop followed by /start, so the scenario ends the outage when it is ready rather than guessing in advance how long a provider needs to notice one. No step in any language reaches it, and Go dropped its binding deliberately. The docstring being removed also asserted the endpoint "is required of every backend", which was true of the document it was written against and is now false. A binding nothing can call is dead surface that misreports the contract. What would bring it back is recorded in its place: a @caching scenario asserting what a stale provider serves *during* an outage needs the flag-state preservation /restart has and /stop + /start does not. Also aligns HttpControl's documentation with the other two changes at that revision -- every state-changing endpoint owes the caller that the new state is being served before it returns, and for /change that promise is about the backend, with the provider's detection latency the business of the event timeout. Neither adds a wait: the class now says why a settle would be the wrong instrument, and where one genuinely belongs when a backend breaks the promise. Signed-off-by: Simon Schrottner --- .../contrib/tools/tck/httpcontrol.py | 84 ++++++++++++------- .../tests/test_http_control.py | 30 ++++--- 2 files changed, 70 insertions(+), 44 deletions(-) diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py index ab1a9ec1a..2f6ef6655 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/httpcontrol.py @@ -9,7 +9,7 @@ import urllib.parse import urllib.request -from .control import BackendControl, ConnectionControl +from .control import BackendControl, ConnectionControl, ControlApi __all__ = [ "DEFAULT_CONFIGURATION", @@ -89,6 +89,16 @@ class HttpControl: resets flag state at the cost of a process restart. The fallback is probed once and remembered for the rest of the suite. + **No settle after a control call, ever.** Every state-changing endpoint -- + ``/start``, ``/change``, ``/reset`` -- owes the caller that the new state is + being served before it returns. A fixed delay here would buy silence rather + than correctness: it is un-tunable, because the window it covers is a + property of the backend and not of this client, and it hides the defect from + the one consumer positioned to notice. Where an adopter is stuck with a + backend that breaks the promise, the wait belongs in *that adoption*, set + explicitly and citing the defect, so that it disappears when the backend is + fixed instead of being inherited by every future adopter from here. + **After a disconnect, ``/start`` rather than ``/reset``.** ``/reset`` is specified to restore flag state, not to bring a stopped backend back up, so a disconnect is recorded and the scenario that follows one is prepared with @@ -103,7 +113,7 @@ def __init__( self, base_url: str, *, - configuration: str = DEFAULT_CONFIGURATION, + backend_configuration: str = DEFAULT_CONFIGURATION, timeout: float = DEFAULT_TIMEOUT, ) -> None: """Build a control for the backend whose control API is rooted at ``base_url``. @@ -112,9 +122,12 @@ def __init__( ``http://localhost:32768``. It must be built from the dynamically mapped host port of the control service, discovered after the stack is up -- a stack under test must not pin host ports. - :param configuration: the named flag configuration to seed. Defaults to - :data:`DEFAULT_CONFIGURATION`, the only name every backend must - support and the one serving the canonical flag set. + :param backend_configuration: the named flag configuration the backend + under test seeds. Defaults to :data:`DEFAULT_CONFIGURATION`, the + only name every backend must support and the one serving the + canonical flag set. Named for the backend because a report's + ``provider.configuration`` is a different thing entirely -- which + mode of the provider was tested. :param timeout: seconds bounding a single control-API request. """ parsed = urllib.parse.urlsplit(base_url) @@ -127,7 +140,7 @@ def __init__( raise ValueError(msg) self._base_url = base_url.rstrip("/") - self._configuration = configuration + self._backend_configuration = backend_configuration self._timeout = timeout self._lock = threading.Lock() @@ -138,14 +151,12 @@ def __init__( self._backend_maybe_down = False @property - def control_api(self) -> str: + def control_api(self) -> ControlApi: """Report that this control drives its backend over the HTTP control API. - The optional property ``BackendControl`` documents. This is the one - control in the package that can answer it without qualification: every - operation below is an HTTP request to ``control-api.yaml``. Leaving it - unsaid would put a report from the normative control path on the same - footing as one from a control that declined to say which path it took. + Every operation below is an HTTP request to ``control-api.yaml``, so + this is the one control in the package that can answer without + qualification. """ return "http" @@ -248,7 +259,18 @@ def prepare_scenario(self) -> None: self._reset_supported = True def change_flag(self) -> None: - """Mutate flag configuration so a conforming provider observes a change.""" + """Mutate flag configuration so a conforming provider observes a change. + + ``/change`` must not return until the new value is actually being + served, and that promise is about the **backend**: once this returns, a + fresh evaluation against the backend resolves the new value. How long + the *provider under test* takes to notice is a property of its transport + -- streaming sees it in milliseconds, a poller may need most of an + interval -- and that is what the suite's event timeout is for. There is + deliberately no wait here: a backend that returns before it serves the + new value makes the provider's detection latency unmeasurable, because + the clock would start before there is anything to detect. + """ self._require("/change") def disconnect(self) -> None: @@ -271,27 +293,25 @@ def reconnect(self) -> None: """ self._start() - def restart(self, seconds: int) -> None: - """Take the backend down for ``seconds`` and bring it back. - - Part of the control API rather than of :class:`~.control.ConnectionControl`: - no scenario drives a bounded outage today, because - :meth:`disconnect`/:meth:`reconnect` let a scenario end the outage when - it is ready instead of guessing how long a provider needs to notice one. - Exposed because the operation is required of every backend and an - adopting suite may want it for its own tests. - - Unlike ``/stop`` followed by ``/start``, this preserves flag state - across the outage. - """ - with self._lock: - self._backend_maybe_down = True - self._require("/restart", {"seconds": str(seconds)}) - with self._lock: - self._backend_maybe_down = False + # NO BINDING FOR ``POST /restart`` + # + # The endpoint simulates a *bounded* outage, and it is ``[OPTIONAL]`` in + # ``control-api.yaml`` because no shipped scenario reaches it. The + # disconnect/reconnect scenario is written as an unbounded outage -- "the + # connection is lost", then "the connection is restored" -- which is + # :meth:`disconnect` followed by :meth:`reconnect`, so a scenario ends the + # outage when it is ready rather than guessing in advance how long the + # provider needs to notice one. + # + # A binding nothing can call is dead surface that also misreports the + # contract, by implying every backend under test owes the endpoint. Go's + # client left it out for the same reason. What would bring it back is + # written down: a ``@caching`` scenario asserting what a stale provider + # serves *during* an outage needs the flag-state preservation that + # ``/restart`` has and ``/stop`` + ``/start`` does not. def _start(self) -> None: - self._require("/start", {"config": self._configuration}) + self._require("/start", {"config": self._backend_configuration}) with self._lock: self._backend_maybe_down = False diff --git a/tools/openfeature-tck/tests/test_http_control.py b/tools/openfeature-tck/tests/test_http_control.py index fc72e4248..97bd5ef0d 100644 --- a/tools/openfeature-tck/tests/test_http_control.py +++ b/tools/openfeature-tck/tests/test_http_control.py @@ -183,19 +183,13 @@ def test_start_names_the_configuration_under_test() -> None: assert ("POST", "/start", "config=default") in stub.requests -def test_a_custom_configuration_is_carried_through() -> None: +def test_a_custom_backend_configuration_is_carried_through() -> None: with _StubControlApi({"/reset": 404}) as stub: - HttpControl(stub.base_url, configuration="ssl").prepare_scenario() + HttpControl(stub.base_url, backend_configuration="ssl").prepare_scenario() assert ("POST", "/start", "config=ssl") in stub.requests -def test_restart_carries_the_outage_duration(stub: _StubControlApi) -> None: - HttpControl(stub.base_url).restart(7) - - assert ("POST", "/restart", "seconds=7") in stub.requests - - def test_change_flag_posts_to_change(stub: _StubControlApi) -> None: HttpControl(stub.base_url).change_flag() @@ -242,16 +236,28 @@ def test_a_trailing_slash_does_not_produce_a_double_slash_path() -> None: def test_the_control_reports_which_api_it_drives_the_backend_through() -> None: - """The optional property ``BackendControl`` documents, answered here. + """The required ``BackendControl`` member, answered here without qualification. - A control that stays quiet has the field omitted from its report, which puts - the normative HTTP path on the same footing as one that declined to say. This - control can say, so it does. + Every operation on this class is an HTTP request to the normative control + API, so this is the one control that can answer the question flatly. A + report whose ``backend.controlApi`` says otherwise for a provider with a + real backend is claiming something it should not. """ with _StubControlApi() as stub: assert HttpControl(stub.base_url).control_api == "http" +def test_there_is_no_binding_for_restart() -> None: + """``/restart`` is optional in the control API and no scenario reaches it. + + A binding nothing can call would imply every backend under test owes the + endpoint, which is what the specification's own description wrongly claimed + before it was corrected. The disconnect/reconnect scenario is an unbounded + outage: ``disconnect`` then ``reconnect``. + """ + assert not hasattr(HttpControl, "restart") + + # -- waiting for the control API --------------------------------------------- From 503f01802c43207607bf164a134a1a62c35c0e57 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 16:25:37 +0200 Subject: [PATCH 31/46] docs(tck): the CI policy, and four corrections the README had coming Adds the policy item 8 settled: a containerised adoption suite is excluded from the default build and a maintainer runs it by hand before merge. Written down because an exclusion nobody wrote down is indistinguishable from an oversight, which is exactly how this went unnoticed. Two reasons, and the second decides it: Docker, and the fact that a conformance suite's honest output includes failures that are not the provider's -- a canonical flag the backend does not seed yet -- so a gate that must be green cannot hold it and an xfail would blame the provider for the backend. The corrections: - control_api is documented as required, closed and stated rather than guessed, in both the control section and the no-backend one; - the compose table's configuration row is backend_configuration; - there is no /restart binding, and the reason, replacing a bullet that presented the endpoint as part of the contract every backend owes; - the settle paragraph no longer claims flagd-testbed#394 closed the window. It did not -- #394 is open and unmerged, the launchpad still returns from /start as soon as /readyz answers, and the ~40 ms window is real and measured. The reason not to sleep is that control-api.yaml now requires every state-changing endpoint to serve before it returns, and that a fixed delay is un-tunable and hides the defect from the one consumer positioned to notice it. Where an adopter is stuck with such a backend the wait belongs in that adoption, named and citing the defect -- which is what the OFREP adoption's SettledControl is, and what Appendix F now prescribes. The self-test tally was also stale: 190, not 163. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 72 ++++++++++++++++++++++++++++----- 1 file changed, 62 insertions(+), 10 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index fff3c84f8..24b13b9c1 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -94,6 +94,26 @@ different timescales — a streaming provider sees a configuration change in mil polls every 30 seconds may need most of a poll interval. Set it to comfortably exceed your worst-case detection latency, or the suite reports timeouts that are really just impatience. +### Running it in CI + +**A containerised adoption suite is excluded from the default build, and a maintainer runs it by +hand before merge.** That is the policy in all four languages' TCKs, and it is written down here +because an exclusion nobody wrote down is indistinguishable from an oversight. + +Two reasons, and the second is the one that actually decides it: + +- It needs Docker, so it cannot be the thing that fails first for a contributor or a runner that has + none. +- A conformance suite reports what is true of the stack under test, which includes failures that are + not the provider's — a canonical flag the backend does not seed yet, a backend that is behind the + spec revision. Those failures are the report. A gate that must be green cannot hold a suite whose + honest output is red, and an `xfail` to make it green would say the provider is at fault when the + backend is. + +So keep it out of the default test task, give it a task of its own, and run that before you merge. +Both adoptions in this repository do exactly that — `poe test-tck` beside `poe test` — and each +records its current tally in its own README, so a reviewer can tell a new failure from a known one. + ## Adding your own scenarios A provider is rarely only a provider. flagd has `fractional` targeting, another vendor has a @@ -419,6 +439,17 @@ control = HttpControl(f"http://localhost:{container.get_launchpad_port()}") one yourself when you use the Compose harness below: it is handed to you as `tck_backend.control`, already awaited ready. +**A control must say which path it drove the backend through.** `control_api` is a required member of +`BackendControl`, typed `ControlApi` — `Literal["http", "in-process"]` — with no default and no +inference from the control's concrete type. `HttpControl` answers `"http"`, `InProcessControl` +answers `"in-process"`, and a custom control states its own. It is the one fact that decides what +everything else in a report is worth: the same scenarios passing over the control API and passing +through in-process manipulation of a provider that *does* have a backend are not the same claim, and +this is the only field that separates them. Nothing outside a control can tell the two apart, and +every run is one or the other — so an absent value would not be "no claim made" but an unfalsifiable +one. The type is closed, so `"HTTP"` or `"grpc"` is a type error here rather than a conformance +report that fails schema validation somewhere else. + ### The container stack The suite starts it. An adopter used to write the container wrapper — and every adopter wrote the @@ -432,7 +463,7 @@ same one, which is why the flagd adoption alone carried a 122-line `conftest.py` | `backend_service` | no | `"backend"` | the Compose service hosting both the control API and the backend | | `control_port` | no | `8080` | container-internal port of the control API | | `additional_ports` | no | `{}` | extra service to ports, for a stack with more than one service. Resolved through the endpoint by service name | -| `configuration` | no | `"default"` | the configuration name passed to `POST /start` | +| `backend_configuration` | no | `"default"` | the configuration name passed to `POST /start` | | `startup_timeout` | no | `60.0` | seconds to wait for the stack and its control API to become reachable | Those names and defaults are fixed across all four languages' TCKs, so a provider shipped in two of @@ -456,11 +487,21 @@ cannot connect three scenarios later. Startup is a real readiness check rather than a pause: the stack comes up with `docker compose up --wait`, then every declared port is waited on until it accepts a connection, then `HttpControl.await_ready()` probes `GET /healthz` until the control API answers. There is -deliberately **no settle after a control call**. Java had a fixed 50ms one; flagd-testbed#394 makes -`POST /start` block until the flags are evaluable, so the sleep covered a window that no longer -exists — and a suite that sleeps instead of holding the control API to its promise stops being able -to detect when the promise breaks. If dropping it makes an adoption flaky, that is a testbed defect -worth filing, not a sleep worth restoring. +deliberately **no settle after a control call**. Java had a fixed 50ms one, and `control-api.yaml` +now states what makes it the wrong instrument: every state-changing endpoint — `/start`, `/change`, +`/reset` — must not return until the new state is actually being served, so a delay here covers a +window the backend is specified to close, and a suite that sleeps instead of holding the API to that +promise stops being able to detect when the promise breaks. The delay is also un-tunable, because +the window is a property of the backend and not of the harness. + +Backends do still break it — flagd-testbed's launchpad returns from `/start` as soon as `/readyz` +answers, which is roughly 40 ms before the flags are evaluable, and +[flagd-testbed#394](https://github.com/open-feature/flagd-testbed/pull/394) is open and unmerged. A +provider that blocks in `initialize` absorbs that window; a stateless one lands in it. Where you are +stuck with such a backend the wait belongs in **your adoption**, set explicitly and citing the +defect, so it reads as a named workaround for one backend and disappears when the backend is fixed — +see the OFREP adoption's `SettledControl`. It does not belong here, where every future adopter would +inherit it without knowing why. `testcontainers` is an **optional** extra rather than a dependency: @@ -496,8 +537,16 @@ Two of the API's requirements are easy to get wrong: running stack. Container orchestrators assign host ports dynamically and cannot reliably preserve them across a restart, so restarting silently invalidates every provider already pointed at the old port, and the failure looks like a flaky provider. -- **`/start` resets flag state; `/restart` preserves it.** An outage must be observable as a change - in availability, never as a change in flag values. +- **An outage must be observable as a change in availability, never as a change in flag values.** + `reconnect()` is `POST /start` with the configuration already in effect, which restores the same + baseline. +- **There is no binding for `POST /restart`.** It simulates a *bounded* outage and is `[OPTIONAL]` in + `control-api.yaml`, because no shipped scenario reaches it: the disconnect/reconnect scenario is + written as an unbounded outage — "the connection is lost", then "the connection is restored" — + which is `disconnect()` then `reconnect()`, so the scenario ends the outage when it is ready rather + than guessing in advance how long the provider needs to notice one. What would bring the endpoint + back is a `@caching` scenario asserting what a stale provider serves *during* an outage, which + needs the flag-state preservation `/restart` has and `/stop` + `/start` does not. ### Providers with no backend @@ -514,6 +563,9 @@ Connection-dependent scenarios have no meaning without a connection, so a backen simply does not implement `ConnectionControl`, leaves `STALE` and `UNAVAILABLE_INIT` undeclared, and those scenarios are skipped with their reason. +Such a control reports `control_api` as `"in-process"`, and that is the whole reason the field is +required rather than guessed: the allowance is only narrow if a report says when it was taken. + ## Findings Four, all confirmed by running the suite rather than by reading code. @@ -611,10 +663,10 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_lifecycle_steps` | the steps that call the provider directly | the in-memory suites skip `@lifecycle`, so the shutdown, re-initialise and metadata steps are driven against a recording provider instead | | `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | | `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity | -| `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping and the control-API it reports, against a stubbed control API | +| `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping, the control API it reports and the absence of a `/restart` binding, against a stubbed control API | ``` -163 passed, 35 skipped, 2 xfailed +190 passed, 35 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; From 9a228a8c64f1ba0f0c77ee47143692576cc47b23 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 23:31:36 +0200 Subject: [PATCH 32/46] chore(tck): re-pin the spec to ccdb8879 Appendix F gains "Running the suite in CI" and the scenario-authoring constraint on the caching gap. No scenario, no canonical flag and no control API change: re-running the asset sync leaves all nine copied files byte-identical, checked by hash before and after. The vendored assets are gitignored and rebuilt by hatch_build_sync.py, so the pin bump is the whole of the commit. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/spec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/openfeature-tck/spec b/tools/openfeature-tck/spec index 93eb1a58d..ccdb88790 160000 --- a/tools/openfeature-tck/spec +++ b/tools/openfeature-tck/spec @@ -1 +1 @@ -Subproject commit 93eb1a58d2d2ec015acb298c914c5822e7a38dd2 +Subproject commit ccdb88790bb4f4beaef14a182d0c2592feab34b2 From e8ba6efa34512e4ae52648e7faa46e6224611a7d Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 23:32:18 +0200 Subject: [PATCH 33/46] feat(tck): fail a run whose canonical features outgrew a reservation A reserved tag is a name held open for scenarios that do not exist yet, and it is only ever temporary: the specification writes them, the tag starts gating something, and the capability becomes declarable. Until this package follows, TckConfig refuses to let anyone declare it -- so the new scenarios skip, for a capability no adopter is allowed to claim, and the report shows a gap the provider may not have. Appendix F calls that the unclaimable capability, and it has no local symptom: a handful of extra skips in a run that is otherwise green. @targeting was reserved until spec revision 26362f85 gave it three scenarios, so this is not hypothetical. This package already checked the property -- test_declaration asserted that no packaged feature file carries a reserved tag -- but a self-test is read by whoever changes this package, and a reservation expires somewhere else. An adoption that re-pinned the assets and ran the suite saw nothing. So the check moves into the plugin, where it fails an adopter's run, and the self-test keeps the converse half: every declarable capability must be carried by some canonical scenario, which catches a tag added to the enum and never wired to anything. extensions.canonical_tags() reads the tags off the packaged feature files. Tag lines only, which matters more than it sounds: events.feature mentions @caching in a comment, saying where those scenarios will go once they exist, so a scan that read whole files would fail every adoption over a sentence. Recursive, because the shape of that directory is the specification's to change. capability.expired_reservations() is the pure comparison, matching JavaScript's expiredReservations, and neither name is exported from the package -- the adopter-facing answer is RESERVED_CAPABILITIES. The tags come from the packaged assets rather than from what was collected, for two reasons. Only the canonical set can expire a reservation: an adopter's own feature reaching for a reserved tag is a mistake in that file, not news about the specification. And a narrowed run cannot then select its way past the check. What is read off the collection is whether the session runs the conformance suite at all -- the plugin is installed for every pytest run in the environment, and an unrelated test suite has no business failing over the contents of these feature files. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 12 ++ .../contrib/tools/tck/capability.py | 26 +++ .../contrib/tools/tck/extensions.py | 40 +++++ .../openfeature/contrib/tools/tck/plugin.py | 68 +++++++- .../openfeature-tck/tests/test_declaration.py | 163 ++++++++++++++---- 5 files changed, 278 insertions(+), 31 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 24b13b9c1..c1c8d2da3 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -335,6 +335,18 @@ reason, back when both were reserved. mirror of the mistake the set exists to prevent — a capability that can be verified, refused the chance — so `@targeting` moved out of it the moment the specification gave it three. +**A reservation this package has outgrown fails the run.** The reservation expires in the +specification repository and `RESERVED_CAPABILITIES` lives here, so on every run the plugin reads the +tags the packaged feature files actually carry and refuses to continue if one of them is still listed +as reserved. Without that, re-pinning the assets onto a revision that gave `@caching` scenarios would +skip them — for a capability `TckConfig` refuses to let anyone declare — and the only visible trace +would be a few more skips. Appendix F calls that the unclaimable capability. The fix when it fires is +to take the tag out of `RESERVED_CAPABILITIES` and decide, per adoption, whether to declare it. + +Only the canonical set can expire a reservation. A feature file of your own reaching for a reserved +tag is a mistake in that file, not news about the specification, so the check reads the packaged +assets — which also means a `-k` or `--deselect` cannot narrow the run past it. + `@numeric-coercion` deserves a note, because it is the one capability here that **the specification does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of unspecified type or size", and languages **may** differentiate between integers and floats "as idioms diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py index 33c008a8d..19c1b2b4e 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py @@ -2,6 +2,7 @@ from __future__ import annotations +import typing from enum import Enum __all__ = ["Capability"] @@ -363,3 +364,28 @@ def capability_for_tag(tag: str) -> Capability | None: capability means reading them back. """ return _BY_TAG.get(tag) + + +def expired_reservations(tags: typing.Iterable[str]) -> tuple[Capability, ...]: + """Reserved capabilities that the tags handed in turn out to carry. + + A non-empty answer means :data:`RESERVED_CAPABILITIES` is out of date: the + scenarios the tag was being held open for now exist, so the capability can + be verified and an adoption should be allowed -- and required -- to say + whether it has it. + + Leaving the reservation in place instead is the mirror of declaring an + unverified capability, and it is the quieter mistake of the two. Declaring + a reserved capability is refused, so nobody can claim it; the new scenarios + are therefore skipped for a capability an adopter has no way to declare, + and the report says a gap exists where the provider may well have none. + Appendix F names that the unclaimable capability. + + Read off the feature files rather than compared against a second list, + because a reservation expires in the specification repository while this + set lives here. Deduplicated and ordered by tag: the tags arrive from every + scenario of every feature file, and one carried twice is not two expiries. + """ + carried = set(tags) + expired = (c for c in RESERVED_CAPABILITIES if c.tag in carried) + return tuple(sorted(expired, key=lambda capability: capability.tag)) diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py index 83617f2b9..14dafc4be 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py @@ -49,6 +49,7 @@ import importlib.resources import inspect +import re import typing from pathlib import Path @@ -57,6 +58,7 @@ "EXTENSIONS_DIRECTORY", "EXTENSIONS_URI_PREFIX", "canonical_root", + "canonical_tags", "collision_problem", "extension_root", "feature_paths", @@ -69,6 +71,9 @@ _PACKAGE = "openfeature.contrib.tools.tck" +_TAG = re.compile(r"@[\w-]+") +"""One Gherkin tag, as it appears on a tag line.""" + CANONICAL_DIRECTORY = "gherkin" """The packaged directory the canonical feature files live in. @@ -185,6 +190,41 @@ def canonical_root() -> Path | None: return None +def canonical_tags() -> frozenset[str]: + """Every Gherkin tag the packaged canonical feature files carry. + + What a reservation is checked against: a tag is reserved because no + canonical scenario carries it, and this is the set that says whether that + is still true. See :func:`~.capability.expired_reservations`. + + **Tag lines only**, which is what tells a tag apart from the same word + written in prose. ``events.feature`` mentions ``@caching`` in a comment, + saying where those scenarios will go once they exist, so a scan that read + the whole file would report the reservation as expired on the strength of a + sentence about it -- and every adoption would then fail on a sentence. + + Recursive, because the shape of the canonical directory is the + specification's to change: a flat scan would answer "no tags" for a file + one directory down, which is silent under-collection, the failure mode the + rest of this module exists to stop. + + Empty when the assets are not reachable as files -- an installation from a + zipimport, say. Everything built on this then degrades to "cannot tell", + which is the honest answer and never a false accusation. + """ + root = canonical_root() + if root is None or not root.is_dir(): + return frozenset() + + tags: set[str] = set() + for feature in sorted(root.rglob("*.feature")): + for line in feature.read_text(encoding="utf-8").splitlines(): + stripped = line.strip() + if stripped.startswith("@"): + tags.update(_TAG.findall(stripped)) + return frozenset(tags) + + def is_canonical(path: Path) -> bool: """Whether a feature file is one of the packaged canonical ones.""" canonical = canonical_root() diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py index 791111cef..dca1ff74e 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py @@ -1,4 +1,4 @@ -"""The pytest plugin: capability gating, scenario state, and the shared step vocabulary. +"""The pytest plugin: capability gating, scenario state, and the step vocabulary. Registered through the ``pytest11`` entry point, so installing this package is all it takes for the step definitions to be available. pytest-bdd resolves steps @@ -10,19 +10,26 @@ in their own ``conftest.py`` is resolved by the same fixture lookup as one this plugin ships, so their scenarios need no glue and no second harness. See :mod:`~.extensions`. + +It is also where the two things a run must refuse to do quietly are checked: +skipping a scenario whose capability was not declared happens loudly, with the +reason, and a reservation the canonical assets have outgrown fails the run +outright rather than skipping scenarios for a capability nobody may claim. """ from __future__ import annotations import typing +from pathlib import Path import pytest from openfeature import api -from .capability import Capability, capability_for_marker +from .capability import Capability, capability_for_marker, expired_reservations from .compose import ComposeBackend, RunningBackend, run_compose_backend from .config import TckConfig +from .extensions import canonical_tags, is_canonical from .state import TckState # The step modules are registered as plugins in their own right, not merely @@ -52,6 +59,63 @@ def pytest_configure(config: pytest.Config) -> None: ) +def pytest_collection_modifyitems(items: list[pytest.Item]) -> None: + """Fail the run if a canonical feature carries a tag still called reserved. + + A reservation is a name held open for scenarios that do not exist yet, and + it is only ever temporary: the specification writes them, the tag starts + gating something, and the capability becomes declarable. Until this package + follows, declaring it is refused -- so those scenarios are skipped for a + capability an adopter cannot claim, and the report shows a gap the provider + may not have. Appendix F calls that the unclaimable capability, and it has + no local symptom at all, which is why it is checked rather than watched + for: ``@targeting`` was reserved until spec revision ``26362f85`` gave it + three scenarios. + + Refused rather than worked around. Dropping the reservation here instead + would let a run declare a capability against a package that does not know + the tag exists, and the point of the check is that a human re-reads + :data:`~.capability.RESERVED_CAPABILITIES` against the specification. + + Only the canonical set can expire a reservation. An adopter's own feature + reaching for a reserved tag is a mistake in that file rather than news + about the specification, so the tags come from the packaged assets and not + from what was collected -- which also means a narrowed run cannot select + its way past the check. What *is* read off the collection is whether this + session runs the conformance suite at all: the plugin is installed for + every pytest run in the environment, and an unrelated test suite has no + business failing over the contents of these feature files. + """ + if not any(_is_canonical_scenario(item) for item in items): + return + + expired = expired_reservations(canonical_tags()) + if not expired: + return + + named = " ".join(capability.tag for capability in expired) + raise pytest.UsageError( + f"reserved capabilities {named} are carried by the canonical feature " + f"files, so the scenarios they were held open for now exist. Remove " + f"them from RESERVED_CAPABILITIES, so an adoption can declare them and " + f"be held to them -- until then those scenarios are skipped for a " + f"capability nobody is allowed to claim." + ) + + +def _is_canonical_scenario(item: pytest.Item) -> bool: + """Whether a collected node is a scenario from the packaged feature files. + + ``__scenario__`` is what pytest-bdd hangs on the function it generates, and + it is readable at collection without running a fixture. The feature file is + then matched by path rather than by the uri it reports, so the check does + not rest on the derivation in :mod:`~.extensions`. + """ + scenario = getattr(getattr(item, "function", None), "__scenario__", None) + filename = getattr(getattr(scenario, "feature", None), "filename", None) + return bool(filename) and is_canonical(Path(str(filename))) + + @pytest.fixture(scope="session") def tck_backend(compose_backend: ComposeBackend) -> typing.Iterator[RunningBackend]: """The Compose stack under test, started once for the whole session. diff --git a/tools/openfeature-tck/tests/test_declaration.py b/tools/openfeature-tck/tests/test_declaration.py index 162e84959..82f84cee5 100644 --- a/tools/openfeature-tck/tests/test_declaration.py +++ b/tools/openfeature-tck/tests/test_declaration.py @@ -21,7 +21,7 @@ from __future__ import annotations -import re +import types import typing import pytest @@ -36,11 +36,14 @@ KnownDeviation, TckConfig, canonical_root, + plugin, ) from openfeature.contrib.tools.tck.capability import ( capability_for_marker, capability_for_tag, + expired_reservations, ) +from openfeature.contrib.tools.tck.extensions import canonical_tags class _StubControl: @@ -75,25 +78,6 @@ def _config(**overrides: typing.Any) -> TckConfig: return TckConfig(**settings) -def _canonical_tags() -> set[str]: - """Every tag the packaged feature files carry, at any level. - - Read off tag lines only. A tag is the whole of the line it appears on in - Gherkin, which is what tells one apart from the same word written in a - comment -- ``events.feature`` mentions ``@caching`` in prose, saying where - those scenarios will go once they exist. - """ - tags: set[str] = set() - root = canonical_root() - assert root is not None, "the packaged canonical features are not on a filesystem" - for feature in sorted(root.glob("*.feature")): - for line in feature.read_text(encoding="utf-8").splitlines(): - stripped = line.strip() - if stripped.startswith("@"): - tags.update(re.findall(r"@[\w-]+", stripped)) - return tags - - # -- the vocabulary ---------------------------------------------------------- @@ -122,16 +106,18 @@ def test_a_reserved_capability_is_one_no_canonical_scenario_carries() -> None: once: ``@targeting`` gained three scenarios at spec revision ``26362f85`` and moved out of the reserved set, which is what this half is for. - The other half is the converse, and it is the half that catches a tag added - to the enum and never wired to anything: every declarable capability must be - carried by some canonical scenario, or declaring it examines nothing. + This half is now also enforced on every adoption's run, by the plugin, and + not only here -- a self-test of this package is read by whoever changes this + package, and the reservation expires somewhere else. What stays here is the + converse, which is the half that catches a tag added to the enum and never + wired to anything: every declarable capability must be carried by some + canonical scenario, or declaring it examines nothing. """ - carried = _canonical_tags() - for capability in RESERVED_CAPABILITIES: - assert capability.tag not in carried, ( - f"{capability.tag} is no longer reserved: the canonical assets now " - f"carry it, so it can be verified and should be declarable" - ) + carried = canonical_tags() + assert not expired_reservations(carried), ( + "the canonical assets now carry a reserved tag, so it can be verified " + "and should be declarable" + ) for capability in DECLARABLE_CAPABILITIES: assert capability.tag in carried, ( f"{capability.tag} is declarable but no canonical scenario carries " @@ -139,6 +125,125 @@ def test_a_reserved_capability_is_one_no_canonical_scenario_carries() -> None: ) +def test_the_packaged_tags_are_read_off_tag_lines_and_not_out_of_prose() -> None: + """The one way this scan can be wrong, pinned against the real assets. + + ``events.feature`` mentions ``@caching`` in a comment, saying where those + scenarios will go once they exist. A scan that read the whole file rather + than its tag lines would call the reservation expired on the strength of + that sentence, and since the plugin fails a run over an expiry, every + adoption would fail over a sentence. + + The second assertion is what keeps the first from being vacuous: it checks + that the prose mention is still there to be mis-read. + """ + tags = canonical_tags() + assert "@events" in tags, "a tag line is read" + assert "@caching" not in tags, "prose is not" + + root = canonical_root() + assert root is not None, "the packaged canonical features are not on a filesystem" + mentions = [ + feature + for feature in sorted(root.rglob("*.feature")) + if "@caching" in feature.read_text(encoding="utf-8") + ] + assert mentions, "nothing mentions @caching any more, so this proves nothing" + + +def test_a_reservation_expires_when_a_scenario_carries_it() -> None: + """The detection itself, which is all the plugin adds to it. + + Deduplicated, because the tags arrive from every scenario of every feature + file and one carried twice is not two expiries. + """ + reserved = sorted(RESERVED_CAPABILITIES, key=lambda c: c.tag) + + assert expired_reservations(["@events", "@object"]) == () + assert expired_reservations(c.tag for c in reserved) == tuple(reserved) + + first = reserved[0] + assert expired_reservations([first.tag, first.tag, "@events"]) == (first,) + + +def _scenario_item(filename: str) -> typing.Any: + """A collected node shaped the way pytest-bdd shapes one. + + ``__scenario__`` on the generated function, carrying the feature it came + from, which is the only part of a node the check reads. + """ + feature = types.SimpleNamespace(filename=filename) + function = types.SimpleNamespace() + function.__scenario__ = types.SimpleNamespace(feature=feature) + return types.SimpleNamespace(function=function) + + +def _a_canonical_feature() -> str: + root = canonical_root() + assert root is not None, "the packaged canonical features are not on a filesystem" + features = sorted(root.rglob("*.feature")) + assert features, "the packaged canonical features are missing" + return str(features[0]) + + +def test_the_plugin_fails_a_run_over_an_expired_reservation( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """An adopter's run, not merely this package's own tests. + + The self-test above is read by whoever changes this package, and a + reservation expires in the specification repository instead -- so an + adoption that re-pinned the assets and ran the suite would see the new + scenarios skipped, for a capability it is refused permission to declare, + and nothing would say so. This is what says so. + """ + items = [_scenario_item(_a_canonical_feature())] + + # Nothing has expired, which is the state every real run is in, and the + # hook is then silent. + plugin.pytest_collection_modifyitems(items) + + reserved = sorted(RESERVED_CAPABILITIES, key=lambda c: c.tag) + monkeypatch.setattr( + plugin, "canonical_tags", lambda: frozenset(c.tag for c in reserved) + ) + + with pytest.raises(pytest.UsageError) as raised: + plugin.pytest_collection_modifyitems(items) + + message = str(raised.value) + for capability in reserved: + assert capability.tag in message + # Named, because the fix is to edit that set against the specification and + # nothing the run can do stands in for it. + assert "RESERVED_CAPABILITIES" in message + + +def test_the_plugin_leaves_a_session_that_is_not_running_the_suite_alone( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The plugin is installed for every pytest run in the environment. + + Which makes the check's trigger part of its correctness: an unrelated test + suite in a project that happens to depend on this package has no business + failing over the contents of these feature files. So the expiry is only + looked for once a canonical scenario is actually collected -- and a node + that is not a scenario at all, or a scenario from an adopter's own + ``extensions`` directory, is neither. + """ + monkeypatch.setattr( + plugin, + "canonical_tags", + lambda: frozenset(c.tag for c in RESERVED_CAPABILITIES), + ) + + not_a_scenario: typing.Any = types.SimpleNamespace() + + plugin.pytest_collection_modifyitems([]) + plugin.pytest_collection_modifyitems([not_a_scenario]) + plugin.pytest_collection_modifyitems([_scenario_item(__file__)]) + + def test_a_tag_maps_onto_the_capability_it_gates() -> None: """The lookup a reporter outside this package needs, in the tag form. From 6d5eaa1df71bf7d1cd4b4ff7313cf658fc5ffa62 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 23:32:47 +0200 Subject: [PATCH 34/46] docs(tck): point at Appendix F for the CI policy instead of restating it "Running it in CI" was four paragraphs of reasoning about why a conformance suite must not be a required gate, written here in this README's own words -- and written again, differently, in three other languages'. That is precisely how the known-deviation and control-path decisions came to have three answers, so the reasoning now lives once in Appendix F and this section points at it. What stays is the mechanism, which is Python's and belongs with Python: the two poe tasks build.yml reaches, the --ignore that excludes the suite from them, the test-tck task beside them, and the comment above them. Two notes stay too, because they are facts about this repository rather than restatements of the appendix: - Docker is not what decides the exclusion here. tests/e2e needs Docker too, has for years, and still runs in the default build on ubuntu-latest. - --ignore does not import the suite, and mypy in these packages is configured over src alone, so nothing in the default build would notice the suite failing to import against the harness. The appendix asks that an excluded suite keep compiling; pytest --collect-only is the form Python has for that, and the adoption branches now run it. The "Known gaps" entry for caching points at the appendix's list rather than naming the gap and stopping, since that list now carries the constraint a @caching scenario has to be written against. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 46 +++++++++++++++++++++------------ 1 file changed, 30 insertions(+), 16 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index c1c8d2da3..54c0331ef 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -96,23 +96,35 @@ worst-case detection latency, or the suite reports timeouts that are really just ### Running it in CI -**A containerised adoption suite is excluded from the default build, and a maintainer runs it by -hand before merge.** That is the policy in all four languages' TCKs, and it is written down here -because an exclusion nobody wrote down is indistinguishable from an oversight. +**Keep the adoption suite out of the default build, give it a task of its own, and write down that +you did.** Why, and the two ways it goes wrong, are in Appendix F's ["Running the suite in +CI"][appendix-f]. It is not restated here: this section used to carry the reasoning in its own +words, in four languages, and that is where the same decisions came to have three different answers. + +The part that is Python's, and so belongs here — two `poe` tasks and one that CI never calls: + +```toml +[tool.poe.tasks] +test = "pytest tests --ignore=tests/tck" +test-cov = "coverage run -m pytest tests --ignore=tests/tck" +test-tck = "pytest tests/tck" +``` -Two reasons, and the second is the one that actually decides it: +`--ignore` on both of the tasks `build.yml` reaches — it runs `poe cov`, which is `test-cov` plus a +coverage report — and a comment above them saying why, so the exclusion cannot read as an oversight. +Both adoptions in this repository are exactly that, and each records its current tally in its own +README so a reviewer running `poe test-tck` can tell a new failure from a known one. -- It needs Docker, so it cannot be the thing that fails first for a contributor or a runner that has - none. -- A conformance suite reports what is true of the stack under test, which includes failures that are - not the provider's — a canonical flag the backend does not seed yet, a backend that is behind the - spec revision. Those failures are the report. A gate that must be green cannot hold a suite whose - honest output is red, and an `xfail` to make it green would say the provider is at fault when the - backend is. +Two Python-specific notes on top of the appendix: -So keep it out of the default test task, give it a task of its own, and run that before you merge. -Both adoptions in this repository do exactly that — `poe test-tck` beside `poe test` — and each -records its current tally in its own README, so a reviewer can tell a new failure from a known one. +- **Docker is not what decides it here.** `tests/e2e` needs Docker too, has needed it for years, and + still runs in the default build on `ubuntu-latest`. The exclusion rests entirely on the second + reason, that the suite's honest output is red. +- **`--ignore` does not import the suite, so nothing checks that it still would.** `mypy` in these + packages is configured over `src` alone. So the default build also collects the excluded suite + without running it — `pytest tests/tck --collect-only` imports every test module, resolves the + feature files and starts no container — which is the appendix's "keep it compiling" in the form + Python has available. ## Adding your own scenarios @@ -678,7 +690,7 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping, the control API it reports and the absence of a `/restart` binding, against a stubbed control API | ``` -190 passed, 35 skipped, 2 xfailed +194 passed, 35 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; @@ -704,7 +716,9 @@ both. Both declare `@variants`, since an in-memory flag set is keyed by variant *whole* context arrives intact: a provider that forwards the targeting key and silently discards every other attribute passes. That needs either an echo operation on the control API or a second canonical flag whose rule keys on a custom attribute. -- **Caching, hooks and flag metadata** are not covered. +- **Caching, hooks and flag metadata** are not covered. Appendix F's ["Known gaps"][appendix-f] is + the list of record, and it now also carries the constraint a `@caching` scenario has to be written + against. [appendix-a]: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md [appendix-f]: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md From 38909b1ce14eaa438e8f02c48ae97ba7dc4c7c92 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 02:26:18 +0200 Subject: [PATCH 35/46] feat(tck): check every scenario the run collects for a reserved tag The check landed canonical-only, on the reasoning that only the specification can expire a reservation -- an adopter's own feature reaching for a reserved tag is a mistake in that file rather than news about the specification. That reasoning is sound and it answers the wrong question. A reserved capability cannot be declared, because TckConfig refuses it, so the capability gate skips every scenario carrying its tag whichever file the tag is in. An extension scenario tagged @caching is therefore unclaimable from the day it is written: it can never run, it can never be claimed, and the report shows a gap the provider may not have. That is the unclaimable-capability failure Appendix F describes, and it is the failure this check exists to surface. It arrives from the adopter's side instead of from upstream and the consequence is identical, so one check covers both causes as long as the message names both remedies -- drop the tag from RESERVED_CAPABILITIES because the specification has given it scenarios, or rename a tag of your own that reached for a reserved name. Go and Java check every scenario the run collected. This is Python coming into line rather than a local judgement: the divergence was put up for a ruling and the broader rule is what it settled on. Two things stay as they were and both are Python's alone. The canonical half is still read off the packaged feature files rather than off the collected items, because an adopter's selection must not be able to narrow a run past the specification's half; an extension has no equivalent source, since its directory is found from the adopter's own test module, so that half comes from the collection as it does in every other language. And the check still waits for a canonical scenario to be collected before it looks at anything: this package is a pytest11 plugin, so the hook fires for every pytest run in an environment that merely has it installed, and an unrelated test suite has no business failing over the contents of these feature files. Tags are read only from scenarios the uri derivation in extensions claims, so a pytest-bdd suite of the adopter's own sharing the session is left alone too. The tags are the markers pytest-bdd derives from the parsed Gherkin, which is what the capability gate itself reads, so the check and the gate cannot disagree about which scenarios are unclaimable -- and a tag inherited from the feature or the rule is included, where Scenario.tags would miss it. Still never the file text: gherkin/events.feature names @caching inside a Gherkin comment, and a text scan would fail every adoption over a sentence. Verified against a real collection rather than only the stubbed items. An adopter suite whose extensions/vendor.feature carries @caching aborts at collection with exit 4 and that message; renaming the tag collects the same 58 scenarios clean; moving @caching into a Gherkin comment collects clean. Narrowing the check back to the canonical set fails exactly one test, the one that covers the widening. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 32 +++-- .../contrib/tools/tck/capability.py | 35 ++--- .../contrib/tools/tck/extensions.py | 11 +- .../openfeature/contrib/tools/tck/plugin.py | 128 ++++++++++++------ .../openfeature-tck/tests/test_declaration.py | 98 ++++++++++++-- 5 files changed, 226 insertions(+), 78 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 54c0331ef..3e1f31f56 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -347,17 +347,25 @@ reason, back when both were reserved. mirror of the mistake the set exists to prevent — a capability that can be verified, refused the chance — so `@targeting` moved out of it the moment the specification gave it three. -**A reservation this package has outgrown fails the run.** The reservation expires in the -specification repository and `RESERVED_CAPABILITIES` lives here, so on every run the plugin reads the -tags the packaged feature files actually carry and refuses to continue if one of them is still listed -as reserved. Without that, re-pinning the assets onto a revision that gave `@caching` scenarios would -skip them — for a capability `TckConfig` refuses to let anyone declare — and the only visible trace -would be a few more skips. Appendix F calls that the unclaimable capability. The fix when it fires is -to take the tag out of `RESERVED_CAPABILITIES` and decide, per adoption, whether to declare it. - -Only the canonical set can expire a reservation. A feature file of your own reaching for a reserved -tag is a mistake in that file, not news about the specification, so the check reads the packaged -assets — which also means a `-k` or `--deselect` cannot narrow the run past it. +**A scenario carrying a reserved tag fails the run.** `TckConfig` refuses to let anyone declare a +reserved capability, so the gate skips every scenario carrying one — for a capability nobody is +permitted to claim, which leaves a gap in the report that the provider may not have. Appendix F calls +that the unclaimable capability, and nothing else notices it: the run is green and the report is +well-formed. So the plugin refuses to continue, naming the tags. + +Two mistakes end there and the message names both remedies. Either the tag arrived with the canonical +feature files, because the specification wrote the scenarios the reservation was held open for and +this package has not followed — take the tag out of `RESERVED_CAPABILITIES` and decide, per adoption, +whether to declare it. Or it arrived from a feature file of your own under `extensions/`, in which +case pick a tag of your own: a reserved tag gates nothing and can never be declared, so a scenario +carrying one can never run. + +The two halves are read differently, and that is deliberate. The canonical tags come from the +packaged files rather than from the collected run, so a `-k` or `--deselect` cannot narrow a run past +the specification's half; your extensions have no such source — the directory is found from your test +module — so those tags come from what was collected. Both are the parsed Gherkin tags, never the file +text: `gherkin/events.feature` names `@caching` in a `#` comment saying where those scenarios will go +once they exist, and a text scan would fail every adoption over a sentence. `@numeric-coercion` deserves a note, because it is the one capability here that **the specification does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of @@ -690,7 +698,7 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping, the control API it reports and the absence of a `/restart` binding, against a stubbed control API | ``` -194 passed, 35 skipped, 2 xfailed +196 passed, 35 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py index 19c1b2b4e..a70ac52a4 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py @@ -369,22 +369,25 @@ def capability_for_tag(tag: str) -> Capability | None: def expired_reservations(tags: typing.Iterable[str]) -> tuple[Capability, ...]: """Reserved capabilities that the tags handed in turn out to carry. - A non-empty answer means :data:`RESERVED_CAPABILITIES` is out of date: the - scenarios the tag was being held open for now exist, so the capability can - be verified and an adoption should be allowed -- and required -- to say - whether it has it. - - Leaving the reservation in place instead is the mirror of declaring an - unverified capability, and it is the quieter mistake of the two. Declaring - a reserved capability is refused, so nobody can claim it; the new scenarios - are therefore skipped for a capability an adopter has no way to declare, - and the report says a gap exists where the provider may well have none. - Appendix F names that the unclaimable capability. - - Read off the feature files rather than compared against a second list, - because a reservation expires in the specification repository while this - set lives here. Deduplicated and ordered by tag: the tags arrive from every - scenario of every feature file, and one carried twice is not two expiries. + A non-empty answer means some scenario is both unrunnable and unclaimable. + Declaring a reserved capability is refused, so the capability gate skips + every scenario carrying its tag, and the report says a gap exists where the + provider may well have none. Appendix F names that the unclaimable + capability, and it is the quieter mirror of declaring a capability nothing + verifies: nobody can claim the tag, so nothing else about the run changes. + + Two things put a reserved tag on a scenario and this reports only that one + of them happened. Either :data:`RESERVED_CAPABILITIES` is out of date -- + the scenarios the tag was held open for now exist, so the capability can be + verified and an adoption should be allowed, and required, to say whether it + has it -- or an adopter has used a reserved name for a tag of their own. + Telling the two apart is the caller's, because the caller is what knows + where the tags came from: the remedies differ and the consequence does not. + + Compared against tags rather than against a second list, because a + reservation expires in the specification repository while this set lives + here. Deduplicated and ordered by tag: the tags arrive from every scenario + of every feature file, and one carried twice is not two expiries. """ carried = set(tags) expired = (c for c in RESERVED_CAPABILITIES if c.tag in carried) diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py index 14dafc4be..473264570 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py @@ -193,9 +193,14 @@ def canonical_root() -> Path | None: def canonical_tags() -> frozenset[str]: """Every Gherkin tag the packaged canonical feature files carry. - What a reservation is checked against: a tag is reserved because no - canonical scenario carries it, and this is the set that says whether that - is still true. See :func:`~.capability.expired_reservations`. + The specification's half of what a reservation is checked against: a tag is + reserved because no canonical scenario carries it, and this is the set that + says whether that is still true. Read off the packaged files rather than + off a collected run, so a ``-k`` or a ``--deselect`` cannot narrow a run + past it. An adopter's own features have no equivalent source -- the + directory is found from their test module -- so the other half of the check + reads them from what was collected. See + :func:`~.capability.expired_reservations`. **Tag lines only**, which is what tells a tag apart from the same word written in prose. ``events.feature`` mentions ``@caching`` in a comment, diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py index dca1ff74e..1005fa87a 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py @@ -13,8 +13,8 @@ It is also where the two things a run must refuse to do quietly are checked: skipping a scenario whose capability was not declared happens loudly, with the -reason, and a reservation the canonical assets have outgrown fails the run -outright rather than skipping scenarios for a capability nobody may claim. +reason, and a scenario carrying a tag this package still calls reserved fails +the run outright rather than being skipped for a capability nobody may claim. """ from __future__ import annotations @@ -29,7 +29,7 @@ from .capability import Capability, capability_for_marker, expired_reservations from .compose import ComposeBackend, RunningBackend, run_compose_backend from .config import TckConfig -from .extensions import canonical_tags, is_canonical +from .extensions import canonical_tags, is_canonical, uri_for from .state import TckState # The step modules are registered as plugins in their own right, not merely @@ -60,60 +60,110 @@ def pytest_configure(config: pytest.Config) -> None: def pytest_collection_modifyitems(items: list[pytest.Item]) -> None: - """Fail the run if a canonical feature carries a tag still called reserved. + """Fail the run if any scenario it collected carries a reserved tag. A reservation is a name held open for scenarios that do not exist yet, and - it is only ever temporary: the specification writes them, the tag starts - gating something, and the capability becomes declarable. Until this package - follows, declaring it is refused -- so those scenarios are skipped for a - capability an adopter cannot claim, and the report shows a gap the provider - may not have. Appendix F calls that the unclaimable capability, and it has - no local symptom at all, which is why it is checked rather than watched - for: ``@targeting`` was reserved until spec revision ``26362f85`` gave it - three scenarios. - - Refused rather than worked around. Dropping the reservation here instead - would let a run declare a capability against a package that does not know - the tag exists, and the point of the check is that a human re-reads - :data:`~.capability.RESERVED_CAPABILITIES` against the specification. - - Only the canonical set can expire a reservation. An adopter's own feature - reaching for a reserved tag is a mistake in that file rather than news - about the specification, so the tags come from the packaged assets and not - from what was collected -- which also means a narrowed run cannot select - its way past the check. What *is* read off the collection is whether this - session runs the conformance suite at all: the plugin is installed for - every pytest run in the environment, and an unrelated test suite has no - business failing over the contents of these feature files. + a reserved capability cannot be declared -- :class:`~.config.TckConfig` + refuses it. So a scenario carrying one reaches the capability gate below + and is skipped, for a capability nobody is permitted to claim: a question + put and silently withdrawn. Appendix F calls that the unclaimable + capability. The run stays green, the report stays well-formed, and nothing + else here notices, which is why it is checked rather than watched for. + + Two different mistakes end there. The check does not tell them apart, + because the consequence is identical and the message names both remedies: + + * the specification wrote the scenarios the tag was held open for and + :data:`~.capability.RESERVED_CAPABILITIES` has not followed -- + ``@targeting`` was reserved until spec revision ``26362f85`` gave it + three scenarios; + * an extension of the adopter's own used a reserved name for a tag of its + own, which is unclaimable from the day the file is written. + + This once fired for the canonical set alone, on the reasoning that only the + specification can expire a reservation. That much is true and it is beside + the point: an extension scenario carrying a reserved tag can never run and + can never be claimed either, which is exactly the failure being surfaced. + It arrives from the adopter rather than from upstream; the consequence does + not care. Go and Java check every scenario the run collected, and so does + this. + + Refused rather than worked around. Treating the tag as declarable here + would let a run claim a capability against a package that does not know the + tag exists, and the point of the check is that a human re-reads that set + against the specification -- or renames their own tag. + + **The two halves have different sources, and only one of them is + collected.** The canonical tags are read off the packaged files rather than + off the items, so a ``-k`` or a ``--deselect`` cannot narrow a run past the + specification's half. An extension's tags have no such source -- the + directory is found from the adopter's own test module, at the moment + :func:`~.extensions.feature_paths` is called -- so they come from what was + collected, as they do in every other language. In practice this hook is + handed the whole collection before anything is deselected, so a selection + does not narrow that half either; that is pytest's hook order rather than a + promise, and the half that must not be narrowable does not lean on it. + + What is read off the collection either way is whether this session runs the + conformance suite at all. This package is a ``pytest11`` plugin, so the + hook fires for every pytest run in an environment that merely has it + installed, and an unrelated test suite has no business failing over the + contents of these feature files. So the check waits for a canonical + scenario, which nothing but :func:`~.extensions.feature_paths` produces, + and it reads tags only from scenarios this suite is running. Go, Java and + JS have an explicit entry point and no equivalent problem. """ - if not any(_is_canonical_scenario(item) for item in items): + running = False + carried: set[str] = set() + for item in items: + feature = _suite_feature(item) + if feature is None: + continue + running = running or is_canonical(feature) + carried.update(f"@{marker.name}" for marker in item.iter_markers()) + + if not running: return - expired = expired_reservations(canonical_tags()) + expired = expired_reservations(carried | canonical_tags()) if not expired: return named = " ".join(capability.tag for capability in expired) raise pytest.UsageError( - f"reserved capabilities {named} are carried by the canonical feature " - f"files, so the scenarios they were held open for now exist. Remove " - f"them from RESERVED_CAPABILITIES, so an adoption can declare them and " - f"be held to them -- until then those scenarios are skipped for a " - f"capability nobody is allowed to claim." + f"reserved capabilities {named} are carried by scenarios this run " + f"collected, and a reserved capability cannot be declared -- so every " + f"scenario carrying one is skipped for a capability nobody is allowed " + f"to claim, and the report shows a gap the provider may not have. " + f"Either they came with the canonical feature files, and the scenarios " + f"they were held open for now exist: remove them from " + f"RESERVED_CAPABILITIES, so an adoption can declare them and be held " + f"to them. Or they came from a feature file of your own under " + f"extensions/, in which case pick tags of your own: a reserved tag " + f"gates nothing and can never be declared." ) -def _is_canonical_scenario(item: pytest.Item) -> bool: - """Whether a collected node is a scenario from the packaged feature files. +def _suite_feature(item: pytest.Item) -> Path | None: + """The feature file behind a collected node, when this suite is what runs it. ``__scenario__`` is what pytest-bdd hangs on the function it generates, and - it is readable at collection without running a fixture. The feature file is - then matched by path rather than by the uri it reports, so the check does - not rest on the derivation in :mod:`~.extensions`. + it is readable at collection without running a fixture. ``None`` for a node + that is not a scenario at all, and for a pytest-bdd scenario belonging to + some other suite that happens to share the session -- neither is this + suite's to fail. + + Which scenarios are this suite's is decided by + :func:`~.extensions.uri_for`, the same derivation a report identifies a + scenario by, so what is checked and what is recorded cannot disagree about + what the run consisted of. """ scenario = getattr(getattr(item, "function", None), "__scenario__", None) filename = getattr(getattr(scenario, "feature", None), "filename", None) - return bool(filename) and is_canonical(Path(str(filename))) + if not filename: + return None + feature = Path(str(filename)) + return feature if uri_for(feature) is not None else None @pytest.fixture(scope="session") diff --git a/tools/openfeature-tck/tests/test_declaration.py b/tools/openfeature-tck/tests/test_declaration.py index 82f84cee5..857e05b23 100644 --- a/tools/openfeature-tck/tests/test_declaration.py +++ b/tools/openfeature-tck/tests/test_declaration.py @@ -23,6 +23,7 @@ import types import typing +from pathlib import Path import pytest @@ -166,16 +167,25 @@ def test_a_reservation_expires_when_a_scenario_carries_it() -> None: assert expired_reservations([first.tag, first.tag, "@events"]) == (first,) -def _scenario_item(filename: str) -> typing.Any: +def _scenario_item(filename: str, *tags: str) -> typing.Any: """A collected node shaped the way pytest-bdd shapes one. ``__scenario__`` on the generated function, carrying the feature it came - from, which is the only part of a node the check reads. + from, and the tags as the markers pytest-bdd turns them into. Those two are + the whole of what the check reads off a node. + + Markers rather than the scenario's own ``tags``, which is the same choice + the capability gate makes and for the same reason: pytest-bdd applies a + marker for every tag on the scenario, on its feature *and* on its rule, so + a feature-level tag is absent from ``Scenario.tags`` and gates every + scenario in the file regardless. Reading what the gate reads is what keeps + the two from disagreeing about which scenarios are unclaimable. """ feature = types.SimpleNamespace(filename=filename) function = types.SimpleNamespace() function.__scenario__ = types.SimpleNamespace(feature=feature) - return types.SimpleNamespace(function=function) + markers = [types.SimpleNamespace(name=tag.lstrip("@")) for tag in tags] + return types.SimpleNamespace(function=function, iter_markers=lambda: iter(markers)) def _a_canonical_feature() -> str: @@ -196,6 +206,10 @@ def test_the_plugin_fails_a_run_over_an_expired_reservation( adoption that re-pinned the assets and ran the suite would see the new scenarios skipped, for a capability it is refused permission to declare, and nothing would say so. This is what says so. + + The upstream half of the check: the tags come off the packaged files, so + nothing an adopter's run does to its selection gets past it. The half that + arrives from the adopter's own side is below. """ items = [_scenario_item(_a_canonical_feature())] @@ -219,17 +233,82 @@ def test_the_plugin_fails_a_run_over_an_expired_reservation( assert "RESERVED_CAPABILITIES" in message +def test_the_plugin_fails_a_run_over_an_adopters_own_reserved_tag( + tmp_path: Path, +) -> None: + """The same failure, arriving from the adopter's side instead of upstream. + + A reserved capability cannot be declared, so the capability gate skips + every scenario carrying its tag -- an extension's included. That scenario + can never run and can never be claimed, which is precisely what this check + exists to surface, so where the tag came from changes the remedy and not + the consequence. The check used to read the canonical set alone and let + this one through. + + Nothing is monkeypatched here: the tag is reserved for real and it is the + adopter's own scenario carrying it, which is the whole of the case. + """ + reserved = sorted(RESERVED_CAPABILITIES, key=lambda c: c.tag) + items = [ + _scenario_item(_a_canonical_feature()), + _scenario_item( + str(tmp_path / "extensions" / "vendor.feature"), + *(capability.tag for capability in reserved), + ), + ] + + with pytest.raises(pytest.UsageError) as raised: + plugin.pytest_collection_modifyitems(items) + + message = str(raised.value) + for capability in reserved: + assert capability.tag in message + # Both remedies, because the check cannot tell which mistake it caught. + assert "RESERVED_CAPABILITIES" in message, "the upstream remedy is named" + assert "extensions/" in message, "so is the adopter's" + + +def test_a_scenario_this_suite_does_not_run_is_not_read(tmp_path: Path) -> None: + """One pytest session can hold more than this suite. + + A project adopting the TCK may have a pytest-bdd suite of its own, and it + is collected into the same session. Its tags are not this suite's business: + a scenario of theirs tagged ``@caching`` is gated by nothing here, runs + normally, and is claimed by nobody -- so failing their run over it would be + an accusation about a file this package has no say in. + + The discriminator is the uri derivation, which answers for the canonical + assets and for an ``extensions`` directory and for nothing else. + """ + items = [ + _scenario_item(_a_canonical_feature()), + _scenario_item( + str(tmp_path / "features" / "billing.feature"), + *(capability.tag for capability in RESERVED_CAPABILITIES), + ), + ] + + plugin.pytest_collection_modifyitems(items) + + def test_the_plugin_leaves_a_session_that_is_not_running_the_suite_alone( - monkeypatch: pytest.MonkeyPatch, + monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: """The plugin is installed for every pytest run in the environment. Which makes the check's trigger part of its correctness: an unrelated test suite in a project that happens to depend on this package has no business - failing over the contents of these feature files. So the expiry is only - looked for once a canonical scenario is actually collected -- and a node - that is not a scenario at all, or a scenario from an adopter's own - ``extensions`` directory, is neither. + failing over the contents of these feature files. So the check waits for a + canonical scenario to be collected, and nothing but ``feature_paths()`` + produces one of those. A node that is not a scenario at all is not one. + + Neither is an adopter's extension, and that is the one asymmetry worth + stating: an extension's tags are read, but an extension is not what says + the suite is running. Nothing an adopter selects reaches that distinction + anyway -- the hook is handed the whole collection before anything is + deselected -- so this is about a session that genuinely collected no + canonical scenario: somebody else's pytest-bdd suite, in an environment + that merely has this package installed. """ monkeypatch.setattr( plugin, @@ -242,6 +321,9 @@ def test_the_plugin_leaves_a_session_that_is_not_running_the_suite_alone( plugin.pytest_collection_modifyitems([]) plugin.pytest_collection_modifyitems([not_a_scenario]) plugin.pytest_collection_modifyitems([_scenario_item(__file__)]) + plugin.pytest_collection_modifyitems( + [_scenario_item(str(tmp_path / "extensions" / "vendor.feature"))] + ) def test_a_tag_maps_onto_the_capability_it_gates() -> None: From b51855554ddca0cf40fb53d0d75817d75c187022 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 09:52:42 +0200 Subject: [PATCH 36/46] feat(tck): the resolution reason is a claim a provider declares Re-pin the spec submodule to c342461a, which moves every resolution-reason assertion out of evaluation.feature, errors.feature and lifecycle.feature -- all thirteen of them -- and into a new gherkin/reason.feature gated as a whole on @standard-reasons. Requirement 2.2.5 is a SHOULD that lets a provider populate `reason` with one of the listed values "or some other string indicating the semantic reason for the returned flag value". Asserting an exact reason in thirteen places narrowed that into a MUST for every adopter, and bought very little: every canonical flag resolves to a value distinct from the caller's default, so a provider that silently falls back was already caught by the value. So @standard-reasons is a claim rather than an exemption -- a provider saying "I use the standard vocabulary with the standard meanings", and reason.feature is what checks it. A provider that does not declare it loses nothing; its values, variants and error codes are asserted everywhere else, on MUSTs. Add the capability, declarable, beside the other twelve. It is the first whose tag is carried at the feature level rather than per scenario, which the gate handles because it reads pytest markers and pytest-bdd marks a scenario from scenario.tags | feature.tags | rule.tags. Verified by running: with the tag undeclared all nine of its scenarios skip in each in-memory suite. Both self-tests declare it, measured before declaring. InMemoryFlag.resolve reports Reason.STATIC for every flag in the decoded set, and a missing flag and a type mismatch both arrive with reason ERROR beside their error code, so the four rule-less rows and the two error scenarios pass in each. The remaining three compose the tag with @targeting and @disabled-flags, neither declared here, so they skip. Self-tests move from 196 passed / 35 skipped to 208 passed / 41 skipped: nine new scenarios in each of the two conformance suites, six running and three skipped. The thirteen removed assertions changed no scenario count -- they were lines inside scenarios that still exist. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 49 ++++++++- tools/openfeature-tck/spec | 2 +- .../contrib/tools/tck/capability.py | 102 +++++++++++++++++- .../tests/test_controllable_conformance.py | 6 ++ .../tests/test_in_memory_conformance.py | 11 ++ 5 files changed, 160 insertions(+), 10 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 3e1f31f56..6415832c9 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -230,6 +230,7 @@ SKIPPED provider does not declare capability @stale. | `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `Capability.REINITIALIZATION` | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) permits rather than requires | | `Capability.TARGETING` | `@targeting` | resolves a flag differently for a matching evaluation context | +| `Capability.STANDARD_REASONS` | `@standard-reasons` | reports the standard resolution reasons, with the meanings [Appendix F][appendix-f] gives them | | `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | `@lifecycle` and `@events` are deliberately separate, and the split matters in both directions. An @@ -275,8 +276,8 @@ conformant provider for something its author could not fix, and nothing could be [Requirement 2.2.4](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) is a **SHOULD** and `types.md` types the field `variant (string, optional)`, so the suite was asserting a `MUST` neither of them states. Since spec revision `26362f85` the variant assertions -live in one gated Scenario Outline of eight rows; the value and reason assertions stay untagged, -because 2.2.3 makes the value a `MUST`. +live in one gated Scenario Outline of eight rows; the value assertions stay untagged, because 2.2.3 +makes the value a `MUST`. `@targeting` was **reserved and undeclarable** until the same revision, on the reading that targeting is backend evaluation logic and out of scope. The scope argument still holds — its three scenarios do @@ -320,8 +321,38 @@ four `disabled-*` flags mirroring `boolean-flag`, `string-flag`, `integer-flag` exactly, differing only in `state`, and one Scenario Outline of four rows asserts that each resolves to the caller's default. Each row's default differs from the flag's configured value, so a provider that ignores the state is caught on the value alone — 2.2.3, a `MUST`. The rows assert neither the -reason, which would rest on 2.2.5's `SHOULD` and its "some other string", nor the variant, since a -disabled flag has resolved none: `@disabled-flags` and `@variants` deliberately do not compose. +reason, which would rest on 2.2.5's `SHOULD` and its "some other string" — it is pinned in +`gherkin/reason.feature` instead, for a provider that opts into the standard meanings — nor the +variant, since a disabled flag has resolved none: `@disabled-flags` and `@variants` deliberately do +not compose. + +`@standard-reasons` is **a claim, not an exemption**, and it is the one capability whose tag is +carried at the *feature* level. 2.2.5 is a `SHOULD` that goes further than 2.2.4 does: it lets a +provider populate `reason` with one of the listed values *"or some other string indicating the +semantic reason for the returned flag value"*. A provider whose backend reports vendor-specific +reasons is therefore conformant, and asserting an exact reason against it would fail it for something +the specification permits. The suite did exactly that until spec revision `c342461a`, in thirteen +places across `evaluation.feature`, `errors.feature` and `lifecycle.feature`, and it bought very +little: every canonical flag resolves to a value distinct from the caller's default, so a provider +that silently falls back is already caught by the value. + +So the reasons live in `gherkin/reason.feature`, gated as a whole. Declaring the capability is a +provider saying *"I use the standard vocabulary with the standard meanings"*, and that file is what +checks the claim — `STATIC` for a rule-less flag, `TARGETING_MATCH` for a matched rule, `DEFAULT` for +an unmatched one, `DISABLED` for a disabled flag, `ERROR` beside an error code. A provider that does +not declare it **loses nothing**: its values, variants and error codes are asserted everywhere else, +on `MUST` requirements. What the declaration adds is something a report's reader can act on — anyone +building telemetry, dashboards or debugging on `reason` can see that the vocabulary was verified +rather than assumed. `STATIC` for the rule-less rows is the call worth flagging: `types.md` types +`DEFAULT` as *"no dynamic evaluation occurred **or** dynamic evaluation yielded no result"*, so a +provider answering `DEFAULT` there is not defective — it does not use the standard meanings, and +should not declare the tag. + +**Tags compose, and here that is load-bearing.** `TARGETING_MATCH` cannot be observed without +targeting and `DISABLED` cannot be observed unless the backend distinguishes a disabled flag, so two +of the file's scenarios also carry `@targeting` and one also carries `@disabled-flags`. Declaring +`@standard-reasons` alone runs the four `STATIC` rows and the two error scenarios, and skips the +other three with their reason. Untagged scenarios are mandatory and always run. `capabilities` defaults to every *declarable* capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from @@ -698,7 +729,7 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping, the control API it reports and the absence of a `/restart` binding, against a stubbed control API | ``` -196 passed, 35 skipped, 2 xfailed +208 passed, 41 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; @@ -716,6 +747,14 @@ are skipped as well. Neither declares `@disabled-flags` either, for the reason i state reaches the flag set and the SDK's provider never reads it — so its four rows are skipped in both. Both declare `@variants`, since an in-memory flag set is keyed by variant name. +Both declare `@standard-reasons`, and it was measured before it was declared: `InMemoryFlag.resolve` +reports `Reason.STATIC` for every flag in the decoded set, and a missing flag and a type mismatch +both arrive with reason `ERROR` beside their error code, so the four rule-less rows and the two error +scenarios pass in each suite. The remaining three scenarios in `reason.feature` compose the tag with +`@targeting` and `@disabled-flags`, neither of which is declared, so they are skipped in both — which +is the composition working rather than a gap, since a reason cannot be observed without the behaviour +that produces it. + ## Known gaps - **Evaluation context passthrough is verified only for the targeting key.** `targeting-key-flag` diff --git a/tools/openfeature-tck/spec b/tools/openfeature-tck/spec index ccdb88790..c342461aa 160000 --- a/tools/openfeature-tck/spec +++ b/tools/openfeature-tck/spec @@ -1 +1 @@ -Subproject commit ccdb88790bb4f4beaef14a182d0c2592feab34b2 +Subproject commit c342461aa95df9e3b46320dbae65e88e5e8b815a diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py index a70ac52a4..52bc45d0f 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py @@ -80,8 +80,8 @@ class Capability(str, Enum): Declaring it runs one Scenario Outline that asserts the variant for each of the eight flags whose variant name the canonical set fixes. Withholding it - skips those rows with the reason and changes nothing else: the value and - reason assertions live in untagged scenarios, because + skips those rows with the reason and changes nothing else: the value + assertions live in untagged scenarios, because `Requirement 2.2.3 `_ makes the value a **MUST**. @@ -143,8 +143,11 @@ class Capability(str, Enum): alone, which rests on 2.2.3, a **MUST**. The rows assert the value and the absence of an error, and deliberately - **not** the reason: pinning ``DISABLED`` would rest on 2.2.5, a **SHOULD** - that permits "some other string". No variant is asserted either, because a + **not** the reason: pinning ``DISABLED`` here would rest on 2.2.5, a + **SHOULD** that permits "some other string". It is pinned in + ``reason.feature`` instead, which composes this tag with + :attr:`STANDARD_REASONS` so that both must be declared before the reason is + asserted. No variant is asserted either, because a disabled flag has resolved no variant and there is none to name -- so this capability and :attr:`VARIANTS` do not compose, which is why the rows are not part of the variant outline. @@ -280,6 +283,14 @@ class Capability(str, Enum): returned the targeted value would pass the first, and one that refuses to evaluate a rule with no targeting key present is caught by the third. + Two more scenarios carry this tag alongside :attr:`STANDARD_REASONS`, in + ``reason.feature``, asserting ``TARGETING_MATCH`` for the hit and + ``DEFAULT`` for the miss. They need both: a provider with no targeting has + no rule to match, so there is no ``TARGETING_MATCH`` for it to report and + the scenario would fail it for an absence rather than a defect. Declaring + this capability alone leaves them skipped, and changes nothing about the + three above. + Declare it if the backend under test can express that rule and the provider forwards the targeting key. A backend with no targeting at all leaves it undeclared and the three scenarios are skipped with the reason -- which is @@ -287,6 +298,89 @@ class Capability(str, Enum): ``targeting`` member, as this package's own does. """ + STANDARD_REASONS = "standard-reasons" + """Provider reports the standard resolution reasons, with the standard meanings. + + **A claim, not an exemption.** `Requirement 2.2.5 + `_ + is a **SHOULD**, and it goes further than 2.2.4 does: it lets a provider + populate ``reason`` with one of the listed values *"or some other string + indicating the semantic reason for the returned flag value"*. A provider + whose backend reports vendor-specific reasons is therefore conformant, and + asserting an exact reason against it would fail it for something the + specification permits. + + An earlier revision of the suite did exactly that, in thirteen places across + ``evaluation.feature``, ``errors.feature`` and ``lifecycle.feature``, and + Appendix F recorded the narrowing as a deliberate exception. It is not one + any more. It bought very little -- every canonical flag resolves to a value + distinct from the caller's default, so a provider that silently falls back + is already caught by the value assertion, and the reason only said *why* it + failed -- and of the thirteen, five sat beside an error-code assertion that + already carries the **MUST**, while the other eight asserted ``STATIC``, the + one reason the specification genuinely leaves open. + + So the reasons now live in ``reason.feature``, gated as a whole at the + feature level. Declaring this capability is a provider saying "I use the + standard vocabulary with the standard meanings", and that file is what + checks the claim. A provider that does not declare it **loses nothing**: its + values, variants and error codes are asserted everywhere else, on **MUST** + requirements. What the declaration adds is something a report's reader can + act on -- anyone building telemetry, dashboards or debugging on ``reason`` + can see that the vocabulary was verified rather than assumed. + + The meanings are the content of the claim, and they constrain nobody who + does not make it: + + * ``STATIC`` -- the flag was resolved from configuration and carries no + targeting rule; + * ``TARGETING_MATCH`` -- a targeting rule matched the evaluation context; + * ``DEFAULT`` -- a targeting rule exists and did not match; + * ``DISABLED`` -- the flag is disabled in the management system; + * ``ERROR`` -- the evaluation failed, and an error code is reported with it. + + ``STATIC`` for the first row is the call worth flagging. ``types.md`` types + ``DEFAULT`` as *"no dynamic evaluation occurred **or** dynamic evaluation + yielded no result"*, which a rule-less flag satisfies as readily as + ``STATIC`` does -- two providers can disagree here and both conform. A + provider that answers ``DEFAULT`` for a rule-less flag is not defective; it + does not use the standard meanings, and should not declare the tag. + + ``ERROR`` is the row where the suite's subject is blurred, and it is + asserted anyway. The other four rest on `Requirement 1.4.7 + `_, + which makes the SDK propagate the provider's reason -- but only *"in cases of + normal execution"*. Abnormal execution is 1.4.9, a **SHOULD** on the *SDK* to + "indicate an error", and nothing requires the provider's reason to survive. + So a passing ``ERROR`` scenario establishes that what reached the + application is coherent, not that the provider produced it. It is still + worth asserting, because the pair is what carries the meaning: the error + code alone is already covered for every provider by ``errors.feature``, + ungated and on a **MUST**, and the reason alone could have been written by + the SDK. An evaluation reporting ``FLAG_NOT_FOUND`` with reason ``STATIC`` + is incoherent whoever wrote it. + + ``SPLIT``, ``UNKNOWN``, ``CACHED`` and ``STALE`` are not asserted. The first + two have no scenario that produces them; ``CACHED`` needs a repeat + evaluation, which nothing here performs without a configuration change in + between, and belongs behind the reserved :attr:`CACHING`; ``STALE`` needs a + scenario asserting what a provider serves *during* an outage, which is the + same gap. + + **Tags compose, and here that is load-bearing.** ``TARGETING_MATCH`` cannot + be observed without targeting and ``DISABLED`` cannot be observed unless the + backend distinguishes a disabled flag, so those scenarios carry + :attr:`TARGETING` and :attr:`DISABLED_FLAGS` as well. A provider declaring + this capability alone runs the four ``STATIC`` rows and the two error + scenarios, and skips the other three with their reason. + + This is also the first capability whose tag is carried at the **feature** + level rather than on each scenario. pytest-bdd marks a scenario from + ``scenario.tags | feature.tags | rule.tags``, so the gate -- which reads + markers -- sees it on every scenario in the file, and ``Scenario.tags`` + alone would not have. + """ + CACHING = "caching" """Reserved, and **not declarable**. No scenario carries this tag yet.""" diff --git a/tools/openfeature-tck/tests/test_controllable_conformance.py b/tools/openfeature-tck/tests/test_controllable_conformance.py index 06fa78725..108587e33 100644 --- a/tools/openfeature-tck/tests/test_controllable_conformance.py +++ b/tools/openfeature-tck/tests/test_controllable_conformance.py @@ -54,6 +54,11 @@ class inherits it rather than choosing it: ``ControllableInMemoryProvider`` changes only how the flag set is *replaced*, and resolution -- including the fact that ``InMemoryFlag.resolve`` never reads ``state`` -- is still the SDK's. Measured the same way, with the same four failures. + + ``STANDARD_REASONS`` is declared, for the reason given there and on the same + measurement: resolution is the SDK's, so the four ``STATIC`` rows and the two + error scenarios pass here identically. The three that compose the tag with + ``TARGETING`` or ``DISABLED_FLAGS`` skip, since neither is declared. """ control = InProcessControl() return TckConfig( @@ -66,6 +71,7 @@ class inherits it rather than choosing it: ``ControllableInMemoryProvider`` Capability.OBJECT, Capability.VARIANTS, Capability.LARGE_INTEGERS, + Capability.STANDARD_REASONS, }, ) diff --git a/tools/openfeature-tck/tests/test_in_memory_conformance.py b/tools/openfeature-tck/tests/test_in_memory_conformance.py index da63c6efd..aef4a69bc 100644 --- a/tools/openfeature-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-tck/tests/test_in_memory_conformance.py @@ -136,6 +136,16 @@ def tck_config() -> TckConfig: in this provider routes a value through a float. ``VARIANTS`` is declared too: the in-memory flag set is keyed by variant name, so the provider has one to report for every flag and does. + + ``STANDARD_REASONS`` is declared, and it was measured before it was: the six + scenarios this provider can reach all pass. ``InMemoryFlag.resolve`` reports + ``Reason.STATIC`` for every flag in the decoded set, so the four rule-less + rows hold; a missing flag and a type mismatch both arrive with reason + ``ERROR`` beside their error code. The other three scenarios in + ``reason.feature`` compose the tag with ``TARGETING`` and ``DISABLED_FLAGS``, + neither of which is declared here, so they skip with that reason -- which is + the capability working as intended rather than a gap: a reason cannot be + observed without the behaviour that produces it. """ return TckConfig( name="in-memory", @@ -146,6 +156,7 @@ def tck_config() -> TckConfig: Capability.OBJECT, Capability.VARIANTS, Capability.LARGE_INTEGERS, + Capability.STANDARD_REASONS, }, ) From 8221e4a4fff70c6291e07c7fc76b549df8a1f117 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 15:29:20 +0200 Subject: [PATCH 37/46] build(tck): check the spec submodule out at the pin before copying its assets A rebase moves the gitlink and not the submodule's working tree. So a checkout can hold a pin naming one revision and assets on disk from another, and nothing in the build says so. That is not hypothetical. Re-pinning the assets last pass and running the suite copied the *previous* pin's Gherkin over the capability the suite had just been given, because the rebase had moved the gitlink and `poe sync-spec-assets` copies whatever is in the submodule working tree. The only thing that noticed was a self-test comparing the capability enum against the packaged assets -- and that guard fires for exactly one symptom, a declarable capability that no canonical scenario carries. A pin that changes nothing but the content of a scenario passes every guard in this package and still runs the wrong suite. The same root cause in another language ran an entire adoption suite against stale assets and reported byte-identical numbers to the run before it, with nothing failing and nothing warning. So `sync()` now brings the submodule to the revision the superproject's index records before it copies anything, and `poe test` already depends on the sync: the suite cannot run against assets it did not just check out. `git submodule update` after a rebase stops being something an operator has to remember, which is the part that was never going to hold. The guarantee is narrower than Go's and the README says so rather than implying otherwise. These assets reach the package by being copied, so a copy can always be made wrong; what this buys is that the suite cannot run without a fresh sync and a sync cannot succeed against any revision but the pinned one. Go consumes the assets as a nested module out of a read-only, checksum-verified cache and has no second artifact to go stale at all. The two fail differently. The index rather than HEAD, because the index is what the next commit records and so what a run is about to claim it tested against. If the update does not reach the pinned revision -- the commit is not in the local object store and could not be fetched -- nothing is copied and the build stops, naming the likely cause. Nothing is written when the working tree is already at the pin, so the ordinary path touches no git state. Two ways out, both loud. OPENFEATURE_TCK_SPEC_UNPINNED=1 copies whatever is checked out, for drafting a change to the canonical assets before there is a revision to pin; it warns on every sync and names the revision it used. And where the pin cannot be read at all the sync warns and continues: building from an unpacked sdist is the ordinary case, with no repository, no pin and the assets already in the tree, but so is a linked git worktree whose .git file names a path outside the running process's filesystem namespace -- a Windows worktree driven from WSL, where git answers inside the submodule and not in the superproject. The guarantee genuinely is not in force there and the warning is what says so. tests/test_spec_assets.py pins both halves. The decision is checked against a scripted git: a working tree behind the pin is checked out and the checkout confirmed, one already at the pin is left alone, a checkout that does not reach the pin stops the build, and neither warning path issues an update. And the invariant itself is checked against this very checkout -- the submodule HEAD equals the pin -- which is the assertion that would have failed last pass. It skips where no pin is readable, because there the guarantee is off and a pass would say otherwise. `hatch_build_sync.py` joins the mypy file list, and pytest gains `pythonpath` so the tests can import it the way the build hook does. Self-tests move from 208 passed / 41 skipped to 214 passed / 42 skipped: seven new tests, one of which skips here because this checkout's pin is unreadable from the shell the suite runs in. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 223 ++++++++++-------- tools/openfeature-tck/hatch_build_sync.py | 149 +++++++++++- tools/openfeature-tck/pyproject.toml | 11 +- .../openfeature-tck/tests/test_spec_assets.py | 206 ++++++++++++++++ 4 files changed, 494 insertions(+), 95 deletions(-) create mode 100644 tools/openfeature-tck/tests/test_spec_assets.py diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 6415832c9..54f66feec 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -8,7 +8,7 @@ its current contents test: the entry point is options-shaped, so a suite for som provider can join it later instead of a second package duplicating the harness. OpenFeature's central promise is that swapping providers does not change application behaviour. -Nothing verifies that today, and every provider tests differently — so "implements the provider +Nothing verifies that today, and every provider tests differently — so "implements the provider contract" is an unverified claim, and a behavioural difference between two providers is discovered by the application that trips over it. @@ -82,15 +82,15 @@ The TCK owns the whole lifecycle: registering the provider under a suite-scoped events, resetting the backend between scenarios, releasing it at the end. **If you find yourself writing test infrastructure, that is a defect here rather than something for you to work around.** -pytest-bdd generates one test per scenario — and one per row of a Scenario Outline — so failures +pytest-bdd generates one test per scenario — and one per row of a Scenario Outline — so failures name a scenario and `-k` selects one as usual. The feature files and canonical flag set are packaged -inside the distribution, so **adopting this package needs no git submodule** — see +inside the distribution, so **adopting this package needs no git submodule** — see [Where the assets come from](#where-the-assets-come-from). ### Timings `TckConfig.event_timeout` is the knob that matters. Providers observe backend changes on wildly -different timescales — a streaming provider sees a configuration change in milliseconds, one that +different timescales — a streaming provider sees a configuration change in milliseconds, one that polls every 30 seconds may need most of a poll interval. Set it to comfortably exceed your worst-case detection latency, or the suite reports timeouts that are really just impatience. @@ -101,7 +101,7 @@ you did.** Why, and the two ways it goes wrong, are in Appendix F's ["Running th CI"][appendix-f]. It is not restated here: this section used to carry the reasoning in its own words, in four languages, and that is where the same decisions came to have three different answers. -The part that is Python's, and so belongs here — two `poe` tasks and one that CI never calls: +The part that is Python's, and so belongs here — two `poe` tasks and one that CI never calls: ```toml [tool.poe.tasks] @@ -110,8 +110,8 @@ test-cov = "coverage run -m pytest tests --ignore=tests/tck" test-tck = "pytest tests/tck" ``` -`--ignore` on both of the tasks `build.yml` reaches — it runs `poe cov`, which is `test-cov` plus a -coverage report — and a comment above them saying why, so the exclusion cannot read as an oversight. +`--ignore` on both of the tasks `build.yml` reaches — it runs `poe cov`, which is `test-cov` plus a +coverage report — and a comment above them saying why, so the exclusion cannot read as an oversight. Both adoptions in this repository are exactly that, and each records its current tally in its own README so a reviewer running `poe test-tck` can tell a new failure from a known one. @@ -122,8 +122,8 @@ Two Python-specific notes on top of the appendix: reason, that the suite's honest output is red. - **`--ignore` does not import the suite, so nothing checks that it still would.** `mypy` in these packages is configured over `src` alone. So the default build also collects the excluded suite - without running it — `pytest tests/tck --collect-only` imports every test module, resolves the - feature files and starts no container — which is the appendix's "keep it compiling" in the form + without running it — `pytest tests/tck --collect-only` imports every test module, resolves the + feature files and starts no container — which is the appendix's "keep it compiling" in the form Python has available. ## Adding your own scenarios @@ -138,10 +138,10 @@ calls `scenarios()`, and write step definitions for whatever is new in a `confte ``` tests/ -├── conftest.py # your step definitions -├── test_conformance.py # the fixture and the one call, unchanged -└── extensions/ - └── fractional.feature +├── conftest.py # your step definitions +├── test_conformance.py # the fixture and the one call, unchanged +└── extensions/ + └── fractional.feature ``` ```python @@ -155,7 +155,7 @@ from openfeature.contrib.tools.tck import TckState def fractional_splits(tck_state: TckState) -> None: ... ``` -That is the whole of it — **no registration, no option and no new argument**. pytest collects +That is the whole of it — **no registration, no option and no new argument**. pytest collects `conftest.py` on its own, pytest-bdd resolves steps through the fixture system, and the canonical step vocabulary is in scope in your feature file beside your own steps. `tck_state` is the same per-scenario state the canonical steps use, so your scenario runs against the provider the suite @@ -175,14 +175,14 @@ configuring anything. There used to be a second call, `features_path()`, which returned the canonical set on its own. It is **gone**. The two differed by one character at the call site and the shorter one silently dropped the extensions directory, so reaching for it produced a green run over fewer scenarios than the -adopter believed had run — which is the worst failure mode available to a conformance suite, because +adopter believed had run — which is the worst failure mode available to a conformance suite, because nothing is there to notice. `canonical_root()` is the supported way to reach the packaged directory for anything that is not "the scenarios to run". ### Your scenarios cannot stand in for ours Every feature file carries a uri, and it is how a canonical scenario is told from an adopter's: -canonical files are the ones under the `gherkin/` prefix and yours are under `extensions/` — the +canonical files are the ones under the `gherkin/` prefix and yours are under `extensions/` — the prefix Go and JavaScript mount theirs under too, so a consumer holding conformance reports from several languages applies one rule. Neither prefix is this package's to choose: Appendix F identifies a canonical feature by its path *relative to the specification's asset directory*, which @@ -196,7 +196,7 @@ called it, and `extensions.py` reports two cases that derivation cannot rule out This is not hypothetical. Java's suite found that a same-named feature file in a second classpath root *replaced* the canonical one, and the run went green having asked the adopter's questions -instead of the specification's — the worst outcome available to a conformance suite. The Python +instead of the specification's — the worst outcome available to a conformance suite. The Python route to the same place is narrower and just as quiet: pytest-bdd names a feature file by its parent directory joined to its own name, so `extensions/gherkin/errors.feature` arrives under the uri the canonical `errors.feature` already occupies. @@ -207,7 +207,7 @@ Not every provider implements every optional part of the contract. Each scenario optional part carries a Gherkin tag, pytest-bdd turns that tag into a pytest marker, and a provider declares what it supports. -**A scenario whose capability was not declared is reported as skipped, with the reason — never as +**A scenario whose capability was not declared is reported as skipped, with the reason — never as passed.** A conformance suite that quietly goes green on scenarios it did not run is worse than no suite at all, so `pytest.skip` carries the reason into the report: @@ -227,15 +227,15 @@ SKIPPED provider does not declare capability @stale. | `Capability.DISABLED_FLAGS` | `@disabled-flags` | resolves a flag disabled in the management system to the code default | | `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | | `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | -| `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | +| `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `Capability.REINITIALIZATION` | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) permits rather than requires | | `Capability.TARGETING` | `@targeting` | resolves a flag differently for a matching evaluation context | | `Capability.STANDARD_REASONS` | `@standard-reasons` | reports the standard resolution reasons, with the meanings [Appendix F][appendix-f] gives them | -| `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | +| `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | `@lifecycle` and `@events` are deliberately separate, and the split matters in both directions. An SDK dispatches `PROVIDER_READY` around `initialize` for *any* provider, so a provider declaring only -`@events` passes the readiness scenario without demonstrating anything — a `NoOpProvider` passes it +`@events` passes the readiness scenario without demonstrating anything — a `NoOpProvider` passes it identically. Meanwhile a stateless provider has a real initialisation to verify but no event stream of its own to declare `@events` for, and gating on `@events` shut it out of a scenario it should be held to. @@ -249,18 +249,18 @@ again is exercising a choice the specification offers it, so withholding this ca `KnownDeviation` entry. The scenario was untagged until spec revision `fc99d5ac`, on the reading that reverting to the -uninitialized state is observable as exactly one thing — being initialisable again. That inference +uninitialized state is observable as exactly one thing — being initialisable again. That inference does not hold, and asserting it unconditionally reported a permitted choice as a conformance failure. A false failure is the mirror image of a vacuous pass, and this suite cares about both. Reverting the -state is not separately observable either — a provider that reverts but refuses reuse presents -identically to one that did neither — so the gated reuse scenario is the only assertion the +state is not separately observable either — a provider that reverts but refuses reuse presents +identically to one that did neither — so the gated reuse scenario is the only assertion the requirement admits. It is worth keeping for the providers that do offer reuse, because releasing the client on shutdown while leaving an initialised flag set behind is easy to write and leaves the provider evaluating against a closed connection rather than failing outright. One practical note, because it is easy to get wrong: `@reinitialization` **narrows** `@lifecycle` rather than standing beside it. The scenario lives in `lifecycle.feature`, which carries `@lifecycle` -at the feature level, so the scenario inherits it and carries both tags — and the gate skips a +at the feature level, so the scenario inherits it and carries both tags — and the gate skips a scenario when *any* capability gating it is undeclared. Reuse is therefore exercised only by an adoption declaring `Capability.LIFECYCLE` **and** `Capability.REINITIALIZATION`; declaring the latter alone leaves the scenario skipped on `@lifecycle` and the declaration unverified. So a provider that @@ -280,13 +280,13 @@ live in one gated Scenario Outline of eight rows; the value assertions stay unta makes the value a `MUST`. `@targeting` was **reserved and undeclarable** until the same revision, on the reading that targeting -is backend evaluation logic and out of scope. The scope argument still holds — its three scenarios do -not test how a backend evaluates a rule — but the conclusion did not: they exist to show the context +is backend evaluation logic and out of scope. The scope argument still holds — its three scenarios do +not test how a backend evaluates a rule — but the conclusion did not: they exist to show the context reached the backend at all, which is a property of the provider and of nothing else. `targeting-key-flag` is the one flag in the canonical set with a rule, specified by behaviour rather than syntax (resolve `hit` when the targeting key is exactly `5c3d8535-f81a-4478-a6d3-afaa4d51199e`), and a matching context resolving to a different value is what catches a provider that drops the -context — no echo endpoint on the control API required. The three scenarios are the matching context, +context — no echo endpoint on the control API required. The three scenarios are the matching context, the non-matching one and no context at all; the second and third are not padding, since a provider that always returned the targeted value would pass the first and one that refuses to evaluate a rule with no targeting key is caught by the third. @@ -294,20 +294,20 @@ with no targeting key is caught by the third. `@disabled-flags` is gated because it needs two things and only one of them comes for free. The caller's default value is held by the provider, which always has it. What the provider also needs is a **signal** that the flag was disabled, told apart from an ordinary resolution and from a missing -flag — and that belongs to the backend and its protocol. One with no disabled state, or one that +flag — and that belongs to the backend and its protocol. One with no disabled state, or one that answers `FLAG_NOT_FOUND` for a disabled flag, gives the provider nothing to act on. -[Appendix F][appendix-f] draws the line elsewhere — a provider whose backend decides, *"such as one -speaking OFREP, cannot: the server never sees the caller's default, so it has no way to return it"* — +[Appendix F][appendix-f] draws the line elsewhere — a provider whose backend decides, *"such as one +speaking OFREP, cannot: the server never sees the caller's default, so it has no way to return it"* — and what this suite measured does not bear that out. flagd's RPC resolver is a remote evaluator by exactly that description and satisfies the capability: the server answers reason `DISABLED` with no variant and no value, and the resolver substitutes the caller's default locally on that signal. flagd's OFREP endpoint answers the same flag with `{"reason": "DISABLED"}` and no `value` and no -`variant` — the same signal in another envelope — and the Python OFREP provider already falls back to +`variant` — the same signal in another envelope — and the Python OFREP provider already falls back to the caller's default for the absent value. It fails these scenarios for a reason unrelated to architecture, which [its own suite](../../providers/openfeature-provider-ofrep/tests/tck/test_ofrep_conformance.py) records. The discrepancy belongs upstream rather than papered over here; what it changes locally is -only what a withheld declaration may be read as — not necessarily an impossibility, so read the +only what a withheld declaration may be read as — not necessarily an impossibility, so read the adoption's own note for which it was. Withholding still needs no `KnownDeviation`, for the reason every gated capability does: a deviation records a gap in behaviour the provider is *required* to have, and this one is optional. @@ -320,9 +320,9 @@ does for `@numeric-coercion`, and gates it. Since spec revision `009afe06` the c four `disabled-*` flags mirroring `boolean-flag`, `string-flag`, `integer-flag` and `float-flag` exactly, differing only in `state`, and one Scenario Outline of four rows asserts that each resolves to the caller's default. Each row's default differs from the flag's configured value, so a provider -that ignores the state is caught on the value alone — 2.2.3, a `MUST`. The rows assert neither the -reason, which would rest on 2.2.5's `SHOULD` and its "some other string" — it is pinned in -`gherkin/reason.feature` instead, for a provider that opts into the standard meanings — nor the +that ignores the state is caught on the value alone — 2.2.3, a `MUST`. The rows assert neither the +reason, which would rest on 2.2.5's `SHOULD` and its "some other string" — it is pinned in +`gherkin/reason.feature` instead, for a provider that opts into the standard meanings — nor the variant, since a disabled flag has resolved none: `@disabled-flags` and `@variants` deliberately do not compose. @@ -338,14 +338,14 @@ that silently falls back is already caught by the value. So the reasons live in `gherkin/reason.feature`, gated as a whole. Declaring the capability is a provider saying *"I use the standard vocabulary with the standard meanings"*, and that file is what -checks the claim — `STATIC` for a rule-less flag, `TARGETING_MATCH` for a matched rule, `DEFAULT` for +checks the claim — `STATIC` for a rule-less flag, `TARGETING_MATCH` for a matched rule, `DEFAULT` for an unmatched one, `DISABLED` for a disabled flag, `ERROR` beside an error code. A provider that does not declare it **loses nothing**: its values, variants and error codes are asserted everywhere else, -on `MUST` requirements. What the declaration adds is something a report's reader can act on — anyone +on `MUST` requirements. What the declaration adds is something a report's reader can act on — anyone building telemetry, dashboards or debugging on `reason` can see that the vocabulary was verified rather than assumed. `STATIC` for the rule-less rows is the call worth flagging: `types.md` types `DEFAULT` as *"no dynamic evaluation occurred **or** dynamic evaluation yielded no result"*, so a -provider answering `DEFAULT` there is not defective — it does not use the standard meanings, and +provider answering `DEFAULT` there is not defective — it does not use the standard meanings, and should not declare the tag. **Tags compose, and here that is load-bearing.** `TARGETING_MATCH` cannot be observed without @@ -355,61 +355,61 @@ of the file's scenarios also carry `@targeting` and one also carries `@disabled- other three with their reason. Untagged scenarios are mandatory and always run. `capabilities` defaults to every *declarable* -capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from +capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from the default, run the suite, and remove only what your provider genuinely cannot do. Leaving a capability out is the only way to withhold it, and one skip carrying its reason is the whole mechanism: the scenario's tags say what was asked, the declaration says whether it was -claimed, and the skip says why it was not. A capability that cannot hold in a language *at all* — +claimed, and the skip says why it was not. A capability that cannot hold in a language *at all* — `@numeric-coercion` where the language has a single numeric type, `@large-integers` on a 32-bit -accessor — is a property of the SDK rather than of the provider, and +accessor — is a property of the SDK rather than of the provider, and [Appendix F][appendix-f] records it once instead of every report restating it. A reserved capability is documented so the vocabulary has a place for it once scenarios exist, and until then it **must not be declared**. Nothing carries the tag, so declaring it cannot be verified, cannot produce a skip, and tells anyone reading the declaration only that something was claimed and nothing examined. `TckConfig` raises if you name one in `capabilities`, and -`DECLARABLE_CAPABILITIES` excludes them — which is the case that matters, because "every capability +`DECLARABLE_CAPABILITIES` excludes them — which is the case that matters, because "every capability except X" is how a reserved tag gets declared by accident rather than by decision. One implementation's published conformance report asserts `@targeting` and `@caching` for exactly that reason, back when both were reserved. `@caching` is the only reserved tag left. Leaving one reserved once it *has* scenarios would be the -mirror of the mistake the set exists to prevent — a capability that can be verified, refused the -chance — so `@targeting` moved out of it the moment the specification gave it three. +mirror of the mistake the set exists to prevent — a capability that can be verified, refused the +chance — so `@targeting` moved out of it the moment the specification gave it three. **A scenario carrying a reserved tag fails the run.** `TckConfig` refuses to let anyone declare a -reserved capability, so the gate skips every scenario carrying one — for a capability nobody is +reserved capability, so the gate skips every scenario carrying one — for a capability nobody is permitted to claim, which leaves a gap in the report that the provider may not have. Appendix F calls that the unclaimable capability, and nothing else notices it: the run is green and the report is well-formed. So the plugin refuses to continue, naming the tags. Two mistakes end there and the message names both remedies. Either the tag arrived with the canonical feature files, because the specification wrote the scenarios the reservation was held open for and -this package has not followed — take the tag out of `RESERVED_CAPABILITIES` and decide, per adoption, +this package has not followed — take the tag out of `RESERVED_CAPABILITIES` and decide, per adoption, whether to declare it. Or it arrived from a feature file of your own under `extensions/`, in which case pick a tag of your own: a reserved tag gates nothing and can never be declared, so a scenario carrying one can never run. The two halves are read differently, and that is deliberate. The canonical tags come from the packaged files rather than from the collected run, so a `-k` or `--deselect` cannot narrow a run past -the specification's half; your extensions have no such source — the directory is found from your test -module — so those tags come from what was collected. Both are the parsed Gherkin tags, never the file +the specification's half; your extensions have no such source — the directory is found from your test +module — so those tags come from what was collected. Both are the parsed Gherkin tags, never the file text: `gherkin/events.feature` names `@caching` in a `#` comment saying where those scenarios will go once they exist, and a text scan would fail every adoption over a sentence. `@numeric-coercion` deserves a note, because it is the one capability here that **the specification -does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of +does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of unspecified type or size", and languages **may** differentiate between integers and floats "as idioms -dictate" — so no requirement says what a provider must do when a value does not fit the accessor it +dictate" — so no requirement says what a provider must do when a value does not fit the accessor it was asked through. That gap is [open-feature/spec#430](https://github.com/open-feature/spec/issues/430). The rule this tag is tested against is therefore **borrowed, not normative**: lossless coercion is -permitted, lossy coercion must fail — `10.0` requested as an integer must succeed, `0.5` must not. +permitted, lossy coercion must fail — `10.0` requested as an integer must succeed, `0.5` must not. It comes from flagd's [numeric coercion ADR](https://github.com/open-feature/flagd/blob/main/docs/architecture-decisions/numeric-coercion.md), -which is scoped to flagd's own implementations, and the tag carries that name — it was -`@strict-numeric-typing` — because two vocabularies for one observable property is worse than one +which is scoped to flagd's own implementations, and the tag carries that name — it was +`@strict-numeric-typing` — because two vocabularies for one observable property is worse than one borrowed name. **A provider that behaves differently is not violating the specification**, so withholding this capability may be a deliberate choice as readily as a defect. @@ -419,15 +419,15 @@ integer is `10`; `integer-flag` (`10`) requested as a float is `10.0`. Rejecting way to pass the first, and the other two are what stop it. The width of the integer accessor is the related property, and it is a capability of its own because -it belongs to the SDK rather than to the provider. Every language can ask for 2^31 − 1, so that -precision scenario is untagged; only the one asking for 2^53 − 1 carries `@large-integers`. A Python -`int` is unbounded, so a Python provider declares it unless something of its own — a 32-bit field in -its wire format, a float on the way through — narrows the value. +it belongs to the SDK rather than to the provider. Every language can ask for 2^31 − 1, so that +precision scenario is untagged; only the one asking for 2^53 − 1 carries `@large-integers`. A Python +`int` is unbounded, so a Python provider declares it unless something of its own — a 32-bit field in +its wire format, a float on the way through — narrows the value. ### Steps that reach the provider directly Everything the suite asks of a provider goes through an OpenFeature client, as an application's -would — except three steps. `the provider is shut down` and `the provider is initialized again` call +would — except three steps. `the provider is shut down` and `the provider is initialized again` call the provider's own `shutdown()` and `initialize()` on the registered instance, and `the provider metadata name should not be empty` asks it for `get_metadata()`. Going through the SDK would test the registry's bookkeeping as much as the provider, and Appendix B already does that; it @@ -436,7 +436,7 @@ per registration. The registry is not told. The client keeps pointing at the same instance, so an evaluation after re-initialising reaches the very object that was shut down and brought back. When the scenario ends, -the SDK shuts the provider down once more on its own — requirement 2.5.3 makes that second call +the SDK shuts the provider down once more on its own — requirement 2.5.3 makes that second call harmless, and the suite relies on it. A direct call that outlasts `TckConfig.ready_timeout` is given up on and fails its scenario with a message rather than hanging the session. @@ -489,7 +489,7 @@ field exists to prevent. Where the specification permits the choice, withholding definitions never talk to a backend directly, which is why the same Gherkin runs unchanged against a containerised backend and against a provider manipulated in-process. -**If your provider talks to a backend, drive it over the HTTP control API** — the document is +**If your provider talks to a backend, drive it over the HTTP control API** — the document is available as `control_api_spec()`, and `HttpControl` is the client for it. That API is the normative contract for those providers, and it is what makes a conformance claim portable: another language's TCK drives the same endpoints against the same stack and must get the same answers. @@ -503,26 +503,26 @@ one yourself when you use the Compose harness below: it is handed to you as `tck already awaited ready. **A control must say which path it drove the backend through.** `control_api` is a required member of -`BackendControl`, typed `ControlApi` — `Literal["http", "in-process"]` — with no default and no +`BackendControl`, typed `ControlApi` — `Literal["http", "in-process"]` — with no default and no inference from the control's concrete type. `HttpControl` answers `"http"`, `InProcessControl` answers `"in-process"`, and a custom control states its own. It is the one fact that decides what everything else in a report is worth: the same scenarios passing over the control API and passing through in-process manipulation of a provider that *does* have a backend are not the same claim, and this is the only field that separates them. Nothing outside a control can tell the two apart, and -every run is one or the other — so an absent value would not be "no claim made" but an unfalsifiable +every run is one or the other — so an absent value would not be "no claim made" but an unfalsifiable one. The type is closed, so `"HTTP"` or `"grpc"` is a type error here rather than a conformance report that fails schema validation somewhere else. ### The container stack -The suite starts it. An adopter used to write the container wrapper — and every adopter wrote the +The suite starts it. An adopter used to write the container wrapper — and every adopter wrote the same one, which is why the flagd adoption alone carried a 122-line `conftest.py` and a 170-line `suite.py` of it. `ComposeBackend` is the whole declaration: | field | required | default | meaning | | --- | --- | --- | --- | -| `compose_file` | yes | — | path to the Compose file, resolved relative to the package directory | -| `backend_ports` | yes | — | container-internal ports the **provider** connects to. The control port is exposed automatically and must not be listed here | +| `compose_file` | yes | — | path to the Compose file, resolved relative to the package directory | +| `backend_ports` | yes | — | container-internal ports the **provider** connects to. The control port is exposed automatically and must not be listed here | | `backend_service` | no | `"backend"` | the Compose service hosting both the control API and the backend | | `control_port` | no | `8080` | container-internal port of the control API | | `additional_ports` | no | `{}` | extra service to ports, for a stack with more than one service. Resolved through the endpoint by service name | @@ -534,10 +534,10 @@ them writes one Compose file and two declarations against it. `tck_backend` is a session-scoped fixture this package's plugin supplies, and it yields two things: -- `tck_backend.control` — the `HttpControl`, already awaited ready. Hand it to `TckConfig.control`. +- `tck_backend.control` — the `HttpControl`, already awaited ready. Hand it to `TckConfig.control`. One per stack: it remembers whether a disconnect left the backend down, so two suites driving the same backend must share it. -- `tck_backend.endpoint` — `host`, `port(internal)` and `port(internal, service=...)`. This is a +- `tck_backend.endpoint` — `host`, `port(internal)` and `port(internal, service=...)`. This is a **factory argument, not a field**: the mapped ports do not exist until the stack is up, which is why `TckConfig.new_provider` is a factory called once per scenario. @@ -551,18 +551,18 @@ Startup is a real readiness check rather than a pause: the stack comes up with `docker compose up --wait`, then every declared port is waited on until it accepts a connection, then `HttpControl.await_ready()` probes `GET /healthz` until the control API answers. There is deliberately **no settle after a control call**. Java had a fixed 50ms one, and `control-api.yaml` -now states what makes it the wrong instrument: every state-changing endpoint — `/start`, `/change`, -`/reset` — must not return until the new state is actually being served, so a delay here covers a +now states what makes it the wrong instrument: every state-changing endpoint — `/start`, `/change`, +`/reset` — must not return until the new state is actually being served, so a delay here covers a window the backend is specified to close, and a suite that sleeps instead of holding the API to that promise stops being able to detect when the promise breaks. The delay is also un-tunable, because the window is a property of the backend and not of the harness. -Backends do still break it — flagd-testbed's launchpad returns from `/start` as soon as `/readyz` +Backends do still break it — flagd-testbed's launchpad returns from `/start` as soon as `/readyz` answers, which is roughly 40 ms before the flags are evaluable, and [flagd-testbed#394](https://github.com/open-feature/flagd-testbed/pull/394) is open and unmerged. A provider that blocks in `initialize` absorbs that window; a stateless one lands in it. Where you are stuck with such a backend the wait belongs in **your adoption**, set explicitly and citing the -defect, so it reads as a named workaround for one backend and disappears when the backend is fixed — +defect, so it reads as a named workaround for one backend and disappears when the backend is fixed — see the OFREP adoption's `SettledControl`. It does not belong here, where every future adopter would inherit it without knowing why. @@ -605,7 +605,7 @@ Two of the API's requirements are easy to get wrong: baseline. - **There is no binding for `POST /restart`.** It simulates a *bounded* outage and is `[OPTIONAL]` in `control-api.yaml`, because no shipped scenario reaches it: the disconnect/reconnect scenario is - written as an unbounded outage — "the connection is lost", then "the connection is restored" — + written as an unbounded outage — "the connection is lost", then "the connection is restored" — which is `disconnect()` then `reconnect()`, so the scenario ends the outage when it is ready rather than guessing in advance how long the provider needs to notice one. What would bring the endpoint back is a `@caching` scenario asserting what a stale provider serves *during* an outage, which @@ -618,8 +618,8 @@ control the backend in-process, where flag operations are direct manipulations o state. `InProcessControl` is the reference. This is a narrow allowance and the obvious thing to abuse. **A provider with an external backend -must use the control API.** Reaching into an external backend from inside the test process — a -test-only admin client, a shared database handle, a hook inside the provider — produces a suite that +must use the control API.** Reaching into an external backend from inside the test process — a +test-only admin client, a shared database handle, a hook inside the provider — produces a suite that passes while proving nothing, because the path it exercised is not the path the contract describes. Connection-dependent scenarios have no meaning without a connection, so a backend-less control @@ -639,7 +639,7 @@ Four, all confirmed by running the suite rather than by reading code. error code**, where the specification requires the code default and `TYPE_MISMATCH`. The client type-checks with `isinstance(value, int)`, and `bool` is a subclass of `int` in Python. -This is **Python-specific** — the identical scenario passes in every other language's suite, which +This is **Python-specific** — the identical scenario passes in every other language's suite, which is a fair advertisement for having more than one implementation. Tracked as [open-feature/python-sdk#619](https://github.com/open-feature/python-sdk/issues/619). @@ -653,8 +653,8 @@ emitting `PROVIDER_CONFIGURATION_CHANGED`. Python's copies its mapping in the co exposes nothing to change it. Tracked as [open-feature/python-sdk#620](https://github.com/open-feature/python-sdk/issues/620). -Only half the machinery is missing — `AbstractProvider` already supplies -`emit_provider_configuration_changed` — which is why `ControllableInMemoryProvider` here is a small +Only half the machinery is missing — `AbstractProvider` already supplies +`emit_provider_configuration_changed` — which is why `ControllableInMemoryProvider` here is a small subclass rather than a reimplementation, and why it should port back to the SDK as a method. ### 3. The in-memory provider does not coerce numbers @@ -662,7 +662,7 @@ subclass rather than a reimplementation, and why it should port back to the SDK `integral-float-flag` (`10.0`) requested as an integer returns the code default with `TYPE_MISMATCH`, and `integer-flag` (`10`) requested as a float does the same. The provider hands each variant back untouched and the client's type check is `isinstance`-based, so neither lossless direction happens. -The lossy scenario passes — every float is rejected — which is exactly the shortcut the two lossless +The lossy scenario passes — every float is rejected — which is exactly the shortcut the two lossless scenarios exist to catch. This is not a defect: `@numeric-coercion` is optional, and the specification does not define the @@ -677,7 +677,7 @@ reason `STATIC` whatever the state, and `InMemoryProvider._resolve` looks only f all four `disabled-*` flags are served exactly as their enabled counterparts are. `canonical_flag_set` is not where this stops. `_decode_canonical_flags` reads the canonical file's -`"state": "DISABLED"`, validates it against `InMemoryFlag.State` and passes it through faithfully — +`"state": "DISABLED"`, validates it against `InMemoryFlag.State` and passes it through faithfully — the self-tests pin that it reaches exactly those four flags and no others. The state survives decoding and then has no effect. @@ -686,7 +686,7 @@ Measured before the tag was gated: all four rows failed on the value in both in- declares `@disabled-flags` and the four scenarios are skipped with that reason. Unlike finding 3 this is a field the SDK offers and does not honour, which is closer to a defect than -to a choice — but the capability is optional, so the honest report is still a withheld declaration +to a choice — but the capability is optional, so the honest report is still a withheld declaration rather than a `KnownDeviation`. It is not filed against the SDK yet. ## Where the assets come from @@ -694,7 +694,7 @@ rather than a `KnownDeviation`. It is not filed against the SDK yet. The Gherkin feature files, the canonical flag set and the control-API document are **not owned by this repository**. They are the language-agnostic conformance artifacts defined in [open-feature/spec][spec] under `specification/assets/provider-tck/`, and every language's TCK ships -the same ones — which is the only reason a conformance claim means the same thing in Python as it +the same ones — which is the only reason a conformance claim means the same thing in Python as it does in Java. **Adopting this package needs no submodule.** The assets are copied into the wheel and the sdist at @@ -714,6 +714,44 @@ is the one thing this suite exists to prevent. A change goes to [open-feature/sp then bump the submodule pin here. Committing no copies means the spec revision this package targets is recorded by the pin and nowhere else, so the two cannot drift apart unnoticed. +### The checkout is part of the sync, not something you remember + +**A rebase moves the gitlink and not the submodule's working tree.** So a checkout can have a pin +naming one revision and assets on disk from another, with nothing in the build saying so. That is +not hypothetical: a sync after a rebase here copied the previous pin's Gherkin over the capability +the suite had just been given, and the only thing that noticed was a self-test comparing the +capability enum against the assets. That guard fires for one symptom. A pin that changes nothing but +the *content* of a scenario would pass every guard in this package and still run the wrong suite — +which is exactly what happened in another language, where an entire adoption suite ran against stale +assets and reported byte-identical numbers to the run before it. + +So `poe sync-spec-assets` brings the submodule to the pinned revision itself before it copies +anything, and `poe test` depends on the sync. **The suite cannot run against assets it did not just +check out**, and `git submodule update` is no longer something an operator has to remember after a +rebase. If the pinned commit is not reachable, nothing is copied and the build stops with a message +saying so, rather than quietly testing the wrong questions. + +**Which guarantee this is, exactly.** Not "a stale copy is impossible": these assets reach the +package by being *copied*, so a copy can always be made wrong — by hand, or by a sync that never +ran. What the wiring buys is narrower and worth stating in its own words — **the suite cannot run +without a fresh sync, and a sync cannot succeed against any revision but the pinned one.** Go's TCK +has the stronger property without doing anything, because it consumes the assets as a nested Go +module out of a read-only, checksum-verified module cache: there is no second artifact to go stale +and the only way past it is a visible `replace` line. The two fail differently, so it is worth +knowing which one you have. + +Two escape hatches, both loud: + +- `OPENFEATURE_TCK_SPEC_UNPINNED=1` copies whatever is checked out in `spec/`, for drafting a change + to the canonical assets before there is a revision to pin. It warns on every sync and names the + revision it used. +- Where the pin cannot be read at all, the sync warns and continues. An unpacked sdist is the + ordinary case — no repository, no pin, and the assets are already in the tree. The other is a + linked git worktree whose `.git` file names a path outside the running process's filesystem + namespace, such as a Windows worktree driven from WSL: git answers inside the submodule and not in + the superproject. The guarantee is genuinely off there, which is what the warning says; run the + sync from a shell that can see the superproject. + This mirrors what `openfeature-flagd-api-testkit` already does for the flagd test harness. ## The self-tests @@ -721,37 +759,38 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | Suite | Subject | Why | | --- | --- | --- | | `test_in_memory_conformance` | the SDK's `InMemoryProvider` | reference adoption for a backend-less provider | -| `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | -| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set is decoded from `canonical-flags.json` — every flag served under the file's own default variant, with the Python type the file wrote, and `state` reaching exactly the four `disabled-*` flags | +| `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | +| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set is decoded from `canonical-flags.json` — every flag served under the file's own default variant, with the Python type the file wrote, and `state` reaching exactly the four `disabled-*` flags | | `test_lifecycle_steps` | the steps that call the provider directly | the in-memory suites skip `@lifecycle`, so the shutdown, re-initialise and metadata steps are driven against a recording provider instead | | `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | | `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity | | `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping, the control API it reports and the absence of a `/restart` binding, against a stubbed control API | +| `test_spec_assets` | where the conformance assets came from | the submodule checkout and the copy are the only thing standing between a run and the wrong questions, and a stale copy is invisible in a pass or a fail | ``` -208 passed, 41 skipped, 2 xfailed +214 passed, 42 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; `test_extensions` takes most of the rest, because the properties it checks are properties of a whole pytest session and it runs a generated adoption in a subprocess to check them. -Neither in-memory suite declares `@lifecycle`, so the six lifecycle scenarios — three about -initialisation, three about shutdown — are skipped in both. That is the point: with no backend to -reach, the initialisation ones would pass without testing anything — which is what they did while the +Neither in-memory suite declares `@lifecycle`, so the six lifecycle scenarios — three about +initialisation, three about shutdown — are skipped in both. That is the point: with no backend to +reach, the initialisation ones would pass without testing anything — which is what they did while the feature was gated on `@events`. Neither declares `@numeric-coercion` either, for the reason in finding 3, so its three scenarios are skipped too. Neither declares `@targeting`: both resolve the same decoded flag set, and `canonical_flag_set` deliberately ignores `targeting-key-flag`'s rule rather than becoming a second implementation of somebody else's evaluator, so those three scenarios -are skipped as well. Neither declares `@disabled-flags` either, for the reason in finding 4 — the -state reaches the flag set and the SDK's provider never reads it — so its four rows are skipped in +are skipped as well. Neither declares `@disabled-flags` either, for the reason in finding 4 — the +state reaches the flag set and the SDK's provider never reads it — so its four rows are skipped in both. Both declare `@variants`, since an in-memory flag set is keyed by variant name. Both declare `@standard-reasons`, and it was measured before it was declared: `InMemoryFlag.resolve` reports `Reason.STATIC` for every flag in the decoded set, and a missing flag and a type mismatch both arrive with reason `ERROR` beside their error code, so the four rule-less rows and the two error scenarios pass in each suite. The remaining three scenarios in `reason.feature` compose the tag with -`@targeting` and `@disabled-flags`, neither of which is declared, so they are skipped in both — which +`@targeting` and `@disabled-flags`, neither of which is declared, so they are skipped in both — which is the composition working rather than a gap, since a reason cannot be observed without the behaviour that produces it. @@ -759,7 +798,7 @@ that produces it. - **Evaluation context passthrough is verified only for the targeting key.** `targeting-key-flag` resolves differently for a matching context, so the `@targeting` scenarios catch a provider that - drops the context — no echo operation needed for that. What is still unverified is that the + drops the context — no echo operation needed for that. What is still unverified is that the *whole* context arrives intact: a provider that forwards the targeting key and silently discards every other attribute passes. That needs either an echo operation on the control API or a second canonical flag whose rule keys on a custom attribute. diff --git a/tools/openfeature-tck/hatch_build_sync.py b/tools/openfeature-tck/hatch_build_sync.py index 054f5ca65..5c7c93431 100644 --- a/tools/openfeature-tck/hatch_build_sync.py +++ b/tools/openfeature-tck/hatch_build_sync.py @@ -8,16 +8,37 @@ package was built against is recorded by the submodule pin and nowhere else, so the two cannot drift apart unnoticed. An *adopter* installing the wheel still needs no submodule: the copies are inside the distribution. + +Copying is the second half of the job. The first is making sure the submodule +working tree is at the revision the pin names, because **a rebase moves the +gitlink and not the working tree** -- see `checkout_pinned_spec`. """ +from __future__ import annotations + +import os import shutil +import subprocess +import warnings +from collections.abc import Callable, Sequence from pathlib import Path ROOT = Path(__file__).parent -SPEC_ASSETS = (ROOT / "spec/specification/assets/provider-tck").resolve() +SPEC_DIRNAME = "spec" +SPEC_ROOT = (ROOT / SPEC_DIRNAME).resolve() +SPEC_ASSETS = (SPEC_ROOT / "specification/assets/provider-tck").resolve() PACKAGE_REL = Path("src/openfeature/contrib/tools/tck") DEST_BASE = ROOT / PACKAGE_REL +UNPINNED_ENV = "OPENFEATURE_TCK_SPEC_UNPINNED" +"""Set to run against whatever is checked out in the submodule, pin or no pin. + +For the one workflow that legitimately wants it: drafting a change to the +canonical assets in a local spec checkout before there is a revision to pin. It +warns on every sync, and it names the revision actually in use, because a suite +running against assets nobody can identify must not be quiet about it. +""" + DO_NOT_EDIT = ( "Generated by hatch_build_sync.py from the open-feature/spec submodule.\n" "DO NOT EDIT. Changes belong in open-feature/spec under\n" @@ -35,8 +56,134 @@ TREES = [("gherkin", "gherkin"), ("flags", "flag_data")] FILES = [("openapi/control-api.yaml", "control-api.yaml")] +GitRunner = Callable[[Sequence[str], Path], str | None] +"""Runs a git command in a directory and returns its output, or ``None`` if it failed. + +A seam, so the decision `checkout_pinned_spec` makes can be tested without a +fixture repository on disk. Failure is ``None`` rather than an exception because +every caller here treats "git could not answer" as a state to report rather than +a crash: building from an unpacked sdist has no repository at all. +""" + + +def _run_git(args: Sequence[str], cwd: Path) -> str | None: + try: + completed = subprocess.run( # noqa: S603 + ["git", *args], # noqa: S607 + cwd=cwd, + capture_output=True, + check=True, + text=True, + ) + except (OSError, subprocess.CalledProcessError): + return None + return completed.stdout.strip() + + +def pinned_revision(git: GitRunner = _run_git) -> str | None: + """The spec revision the superproject's index records for the submodule. + + The index rather than ``HEAD``, because the index is what the next commit + will record and so what a run is about to claim it tested against. ``None`` + when there is no repository to ask, which is ordinary: an sdist has none. + """ + entry = git(["ls-files", "-s", "--", SPEC_DIRNAME], ROOT) + if not entry: + return None + fields = entry.split() + # "160000 0spec". Anything else is not a gitlink -- a plain + # directory checked in under that name, most likely -- and there is no pin. + if len(fields) < 2 or fields[0] != "160000": + return None + return fields[1] + + +def checkout_pinned_spec(git: GitRunner = _run_git) -> str | None: + """Bring the submodule working tree to the revision the pin names. + + **This is the reason this function exists: a rebase moves the gitlink and + not the submodule's working tree.** So a checkout can have a pin saying one + revision and assets on disk from another, and nothing about the build says + so. That is not hypothetical -- it happened here, the copied Gherkin was one + pin behind the capability the suite was declaring, and a self-test of the + enum against the assets is what caught it. That guard fires for one symptom. + A pin that changes only the *content* of a scenario would pass every guard + in this package and still run the wrong suite, which is what happened in + another language: a whole adoption suite ran against stale assets and + reported byte-identical numbers to the previous pass, and nothing failed and + nothing warned. + + So the checkout is part of the copy rather than something the operator is + expected to remember, and the copy is a dependency of the test task. The + suite cannot run against assets it did not just fetch. + + Returns the pinned revision, or ``None`` when there is no pin to read. + + **When the pin cannot be read this warns rather than failing.** Building + from an unpacked sdist is the ordinary case -- no repository, no pin, and + the assets are already in the tree. The other case is a git checkout whose + superproject this process cannot reach, which happens for a linked worktree + whose ``.git`` file names a path in another filesystem namespace: git inside + the submodule answers, git in the superproject does not. Warning rather than + failing keeps that environment usable; the warning is what says the + guarantee is not in force, and the remedy is to run the sync from a shell + that can see the superproject. + """ + if os.environ.get(UNPINNED_ENV): + head = git(["rev-parse", "HEAD"], SPEC_ROOT) + warnings.warn( + f"{UNPINNED_ENV} is set: syncing the conformance assets from whatever " + f"is checked out in {SPEC_ROOT} ({head or 'unknown revision'}) rather " + f"than from the revision the submodule pin names. Nothing records " + f"which questions this run asked", + stacklevel=2, + ) + return None + + pinned = pinned_revision(git) + if pinned is None: + if SPEC_ROOT.exists(): + warnings.warn( + f"could not read the submodule pin for {SPEC_DIRNAME} from a git " + f"index in {ROOT}, so the conformance assets are being copied from " + f"the submodule working tree unverified. If this is a checkout " + f"rather than an unpacked sdist, a rebase may have moved the pin " + f"without moving that working tree: run `git submodule update " + f"--init tools/openfeature-tck/spec` from a shell that can see the " + f"superproject, then sync again", + stacklevel=2, + ) + return None + + head = git(["rev-parse", "HEAD"], SPEC_ROOT) + if head == pinned: + return pinned + + print( # noqa: T201 + f"spec submodule is at {head or '(not checked out)'}, the pin names " + f"{pinned}: checking out the pinned revision" + ) + git(["submodule", "update", "--init", "--", SPEC_DIRNAME], ROOT) + + head = git(["rev-parse", "HEAD"], SPEC_ROOT) + if head != pinned: + msg = ( + f"the {SPEC_DIRNAME} submodule is at {head or '(not checked out)'} and " + f"the pin names {pinned}, and `git submodule update --init` did not " + f"move it. The assets would be from the wrong revision, so nothing is " + f"copied. Most likely the pinned commit is not in the local object " + f"store and could not be fetched; check network access to " + f"open-feature/spec, or fetch it by hand. To copy from the working " + f"tree anyway -- for drafting a spec change that has no revision yet " + f"-- put {UNPINNED_ENV}=1 in the environment." + ) + raise RuntimeError(msg) + return pinned + def sync() -> None: + checkout_pinned_spec() + if not SPEC_ASSETS.exists(): msg = ( f"Conformance assets not found at {SPEC_ASSETS}. " diff --git a/tools/openfeature-tck/pyproject.toml b/tools/openfeature-tck/pyproject.toml index efc89fbe5..2cb6f12ff 100644 --- a/tools/openfeature-tck/pyproject.toml +++ b/tools/openfeature-tck/pyproject.toml @@ -73,9 +73,16 @@ artifacts = [ [tool.hatch.build.hooks.custom] +# `hatch_build_sync.py` sits beside this file rather than under src/, because it +# is build machinery rather than package content. It is on the path anyway, so +# the tests that hold its submodule-pin check to its promise can import it the +# way the build hook does. +[tool.pytest.ini_options] +pythonpath = ["."] + [tool.mypy] -mypy_path = "src" -files = ["src", "tests"] +mypy_path = ["src", "."] +files = ["src", "tests", "hatch_build_sync.py"] python_version = "3.10" namespace_packages = true explicit_package_bases = true diff --git a/tools/openfeature-tck/tests/test_spec_assets.py b/tools/openfeature-tck/tests/test_spec_assets.py new file mode 100644 index 000000000..a36ef9181 --- /dev/null +++ b/tools/openfeature-tck/tests/test_spec_assets.py @@ -0,0 +1,206 @@ +"""Where the conformance assets came from, and that it is the revision claimed. + +The assets are not this repository's. They arrive through a submodule, the pin is +the only record of which revision a run asked its questions at, and the copies in +the package are gitignored -- so the one thing that can go wrong silently is the +copies being from a different revision than the pin names. + +It has gone wrong. A rebase moves the gitlink and not the submodule's working +tree, so a sync after a rebase copied the *previous* pin's Gherkin over the +capability the suite had just been given, and only a self-test comparing the enum +against the assets noticed. That guard fires for one symptom -- a declarable +capability no scenario carries. A pin that changes nothing but the content of a +scenario would pass every guard in this package and still run the wrong suite, +which is what happened in another language: an entire adoption suite ran against +stale assets and reported byte-identical numbers to the previous run. + +So the checkout is wired into the copy, the copy is a dependency of the test +task, and this is what holds both ends of that to their promise. +""" + +from __future__ import annotations + +import warnings +from collections.abc import Sequence +from pathlib import Path + +import hatch_build_sync +import pytest +from hatch_build_sync import ( + SPEC_ROOT, + UNPINNED_ENV, + checkout_pinned_spec, + pinned_revision, +) + +PIN = "1111111111111111111111111111111111111111" +OTHER = "2222222222222222222222222222222222222222" + + +class _FakeGit: + """A git that answers from a script and remembers what it was asked. + + Enough to pin the decision without a fixture repository: the three questions + the checkout asks are the gitlink in the index, ``HEAD`` in the submodule, + and the update itself, and what matters is which of them are asked and in + what order. + """ + + def __init__(self, *, pin: str | None, heads: Sequence[str | None]) -> None: + self.pin = pin + self.heads = list(heads) + self.calls: list[list[str]] = [] + + def __call__(self, args: Sequence[str], cwd: Path) -> str | None: + self.calls.append(list(args)) + if args[0] == "ls-files": + return f"160000 {self.pin} 0\tspec" if self.pin else None + if args[0] == "rev-parse": + return self.heads.pop(0) if self.heads else None + return "" + + @property + def updated(self) -> bool: + return any(call[0] == "submodule" for call in self.calls) + + +# -- against the real checkout ------------------------------------------------ + + +def test_the_assets_on_disk_are_the_revision_the_pin_names() -> None: + """The invariant itself, measured against this very checkout. + + ``poe test`` runs the sync first, so by the time this runs the checkout has + already been brought to the pin -- which makes this a check that it really + was, rather than a check of something the sync would have had to do anyway. + It is also the assertion that would have failed in the pass where the stale + Gherkin got through. + + Skipped, loudly, where the pin cannot be read at all. That is an unpacked + sdist, which has no repository and no pin, and a linked git worktree whose + ``.git`` file names a path this process's filesystem namespace cannot follow + -- git inside the submodule answers there and git in the superproject does + not. The guarantee is genuinely not in force in those environments, and a + skip that says so is the honest report; a pass would not be. + """ + pinned = pinned_revision() + if pinned is None: + pytest.skip( + "no submodule pin is readable from here, so which revision these " + "assets came from cannot be established -- see checkout_pinned_spec" + ) + + head = hatch_build_sync._run_git(["rev-parse", "HEAD"], SPEC_ROOT) + assert head == pinned, ( + f"the spec submodule is checked out at {head} and the pin names " + f"{pinned}, so the assets this run tested against are not the ones it " + f"claims. Run `poe sync-spec-assets`" + ) + + +def test_only_a_gitlink_counts_as_a_pin() -> None: + """A plain directory checked in under that name is not a revision. + + Reading the second field of whatever ``ls-files -s`` printed would turn a + blob's hash into a commit id and compare it against the submodule's HEAD + forever after, which fails in a way that explains nothing. + """ + assert pinned_revision(lambda args, cwd: "100644 abc123 0\tspec") is None + assert pinned_revision(lambda args, cwd: "") is None + assert pinned_revision(lambda args, cwd: None) is None + assert pinned_revision(lambda args, cwd: f"160000 {PIN} 0\tspec") == PIN + + +# -- the decision ------------------------------------------------------------- + + +def test_a_working_tree_behind_the_pin_is_checked_out() -> None: + """The case this exists for: a rebase moved the gitlink and nothing else.""" + git = _FakeGit(pin=PIN, heads=[OTHER, PIN]) + + assert checkout_pinned_spec(git) == PIN + + assert git.updated, "the stale working tree was left where it was" + assert git.calls[-1][0] == "rev-parse", ( + "the checkout was not confirmed after being asked for" + ) + + +def test_a_working_tree_already_at_the_pin_is_left_alone() -> None: + """No git writes on the ordinary path. + + Every ``poe test`` runs this, including in a checkout somebody is midway + through something in. Moving a submodule that is already where it should be + is a write nobody asked for. + """ + git = _FakeGit(pin=PIN, heads=[PIN]) + + assert checkout_pinned_spec(git) == PIN + + assert not git.updated + assert [call[0] for call in git.calls] == ["ls-files", "rev-parse"] + + +def test_a_checkout_that_does_not_reach_the_pin_stops_the_build() -> None: + """Refused rather than warned, because here the answer is known and wrong. + + Unlike the unreadable-pin case, nothing is in doubt: the pin says one + revision, the working tree is at another, and the update did not close the + gap -- the commit is probably not in the local object store. Copying now + would produce assets from a revision the build is about to claim it did not + use. + """ + git = _FakeGit(pin=PIN, heads=[OTHER, OTHER]) + + with pytest.raises(RuntimeError) as raised: + checkout_pinned_spec(git) + + message = str(raised.value) + assert PIN in message and OTHER in message + assert "nothing is copied" in message + assert "fetch" in message, "the likely cause is named" + assert UNPINNED_ENV in message, "so is the deliberate way round it" + + +def test_an_unreadable_pin_warns_and_lets_the_build_continue( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Two environments reach this and only one of them is a problem. + + An unpacked sdist has no repository, no pin and no submodule, and the assets + are already in the tree: nothing to say. A checkout whose superproject this + process cannot reach has all three and no way to check them, which is worth + a warning every time -- it is the only signal that the guarantee is off. + """ + git = _FakeGit(pin=None, heads=[]) + + monkeypatch.setattr(hatch_build_sync, "SPEC_ROOT", Path("/no/such/submodule")) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + assert checkout_pinned_spec(git) is None + assert not caught, f"an sdist build has nothing to warn about: {caught}" + + monkeypatch.setattr(hatch_build_sync, "SPEC_ROOT", SPEC_ROOT) + with pytest.warns(UserWarning, match="copied from the submodule working tree"): + assert checkout_pinned_spec(git) is None + assert not git.updated + + +def test_the_escape_hatch_says_which_revision_it_used( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Drafting a spec change locally is legitimate; doing it quietly is not. + + The revision is named because the point of the pin is that a report can say + what it ran against, and a run in this mode cannot. + """ + monkeypatch.setenv(UNPINNED_ENV, "1") + git = _FakeGit(pin=PIN, heads=[OTHER]) + + with pytest.warns(UserWarning, match=OTHER): + assert checkout_pinned_spec(git) is None + + assert not git.updated + assert [call[0] for call in git.calls] == ["rev-parse"], ( + "the pin is not consulted, which is the whole of what the flag does" + ) From 691c68b4be91a07cb6c66e3634c0e5bd7bdc4617 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 15:29:20 +0200 Subject: [PATCH 38/46] feat(tck): refuse a capability this SDK cannot express, rather than leaving it to adopters Re-pin the spec submodule to 89b1519a, which adds a fifth rule for declaring to Appendix F -- a capability the language's SDK cannot express is refused by the implementation, not left to adopters -- and corrects the canonical flag set's comments, which still told every reader that every scenario expects reason STATIC. Neither feature file changes. Two such capabilities exist anywhere: @large-integers where the integer accessor is a 32-bit Integer, and @numeric-coercion where the language has one numeric type and "a float requested as an integer" does not name two different requests. Neither says anything about a provider. Leaving it to adopters means every adopter in the language has to know a fact about their language and remember to act on it, and in one implementation three separate suites each left the same capability undeclared with its own comment restating the same property -- three places to get right, and a single wrong one puts a claim in a report that no scenario could have verified. So INEXPRESSIBLE_CAPABILITIES maps such a capability to the property of the SDK that puts the question out of reach, TckConfig refuses one at construction, the capability gate skips its scenarios with that reason, and a knownDeviations entry may not name one -- the gap would be the language's and the entry would attribute it to this provider. A mapping rather than a set because the message has to name the property: an adopter who reaches this has done nothing wrong and "the specification says you may not" is not something they can act on. The two refusals stay distinguishable, in separate predicates with separate messages and separate skip reasons. A reservation is global and temporary -- no scenario anywhere carries the tag, and it expires the moment the specification writes one. An inexpressibility is one language's and permanent: the scenarios exist and other languages run and pass them. A reader seeing a capability absent from a report has to be able to tell "this provider declined" from "no provider in this language can be asked", because only the first says anything about the provider. Where a scenario is gated by both kinds, the language-wide reason wins, because the provider's declaration could not have made that scenario run either way. **The mapping is empty in Python, and that was measured rather than assumed.** `int` is arbitrary-precision; FlagType.INTEGER and FlagType.FLOAT are separate, reach separate provider methods and are type-checked against `int` and `float` separately. All four questions the two tags ask were put through the SDK's own client against a provider implementing the borrowed coercion rule: 2^53 - 1 resolved exactly, 0.5 as an Integer gave TYPE_MISMATCH and the caller's default, 10.0 as an Integer gave 10 and 10 as a Float gave 10.0. The adoptions agree from the other direction, and this was measured too rather than reasoned from the source: declaring @numeric-coercion in both flagd suites and running it, the in-process resolver refuses 0.5 as an integer and widens 10 to a float, while the RPC resolver widens 10 and silently narrows 0.5 to 0. Two resolvers of one provider giving different answers to the same three questions is exactly what a language that could not ask them makes impossible. Both are defects in an implementation, withholding the tag is the honest report for each, and neither is anything the language prevents. The third scenario fails on both for a third reason again -- flagd-testbed seeds no integral-float-flag -- which is also why the run leaves both suites' declarations exactly as they were. So nothing here is in force, and the machinery is added anyway. The rule belongs to Appendix F rather than to this package, a future capability may hit it, and the costs are not symmetric: an unused mechanism is a few lines nobody reads, while a missing one is discovered by an adopter publishing a claim no scenario could have examined. It is exercised rather than left dead -- the tests supply an entry and drive the refusal, the deviation refusal, both skip reasons and the precedence between them, so a mechanism with no instances is still known to work. DECLARABLE_CAPABILITIES is derived from both sets rather than listing what it excludes, which matters precisely because the new one is empty here: a derivation that quietly dropped it would look right in Python forever and be wrong in the one language where an entry gets added. Self-tests move from 214 passed / 42 skipped to 221 passed / 42 skipped. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 225 ++++++++++------- tools/openfeature-tck/spec | 2 +- .../openfeature/contrib/tools/tck/__init__.py | 8 +- .../contrib/tools/tck/capability.py | 101 +++++++- .../openfeature/contrib/tools/tck/config.py | 74 +++++- .../openfeature/contrib/tools/tck/plugin.py | 66 ++++- .../openfeature-tck/tests/test_declaration.py | 233 +++++++++++++++++- 7 files changed, 595 insertions(+), 114 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 54f66feec..7bb2ac0a8 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -8,7 +8,7 @@ its current contents test: the entry point is options-shaped, so a suite for som provider can join it later instead of a second package duplicating the harness. OpenFeature's central promise is that swapping providers does not change application behaviour. -Nothing verifies that today, and every provider tests differently — so "implements the provider +Nothing verifies that today, and every provider tests differently — so "implements the provider contract" is an unverified claim, and a behavioural difference between two providers is discovered by the application that trips over it. @@ -82,15 +82,15 @@ The TCK owns the whole lifecycle: registering the provider under a suite-scoped events, resetting the backend between scenarios, releasing it at the end. **If you find yourself writing test infrastructure, that is a defect here rather than something for you to work around.** -pytest-bdd generates one test per scenario — and one per row of a Scenario Outline — so failures +pytest-bdd generates one test per scenario — and one per row of a Scenario Outline — so failures name a scenario and `-k` selects one as usual. The feature files and canonical flag set are packaged -inside the distribution, so **adopting this package needs no git submodule** — see +inside the distribution, so **adopting this package needs no git submodule** — see [Where the assets come from](#where-the-assets-come-from). ### Timings `TckConfig.event_timeout` is the knob that matters. Providers observe backend changes on wildly -different timescales — a streaming provider sees a configuration change in milliseconds, one that +different timescales — a streaming provider sees a configuration change in milliseconds, one that polls every 30 seconds may need most of a poll interval. Set it to comfortably exceed your worst-case detection latency, or the suite reports timeouts that are really just impatience. @@ -101,7 +101,7 @@ you did.** Why, and the two ways it goes wrong, are in Appendix F's ["Running th CI"][appendix-f]. It is not restated here: this section used to carry the reasoning in its own words, in four languages, and that is where the same decisions came to have three different answers. -The part that is Python's, and so belongs here — two `poe` tasks and one that CI never calls: +The part that is Python's, and so belongs here — two `poe` tasks and one that CI never calls: ```toml [tool.poe.tasks] @@ -110,8 +110,8 @@ test-cov = "coverage run -m pytest tests --ignore=tests/tck" test-tck = "pytest tests/tck" ``` -`--ignore` on both of the tasks `build.yml` reaches — it runs `poe cov`, which is `test-cov` plus a -coverage report — and a comment above them saying why, so the exclusion cannot read as an oversight. +`--ignore` on both of the tasks `build.yml` reaches — it runs `poe cov`, which is `test-cov` plus a +coverage report — and a comment above them saying why, so the exclusion cannot read as an oversight. Both adoptions in this repository are exactly that, and each records its current tally in its own README so a reviewer running `poe test-tck` can tell a new failure from a known one. @@ -122,8 +122,8 @@ Two Python-specific notes on top of the appendix: reason, that the suite's honest output is red. - **`--ignore` does not import the suite, so nothing checks that it still would.** `mypy` in these packages is configured over `src` alone. So the default build also collects the excluded suite - without running it — `pytest tests/tck --collect-only` imports every test module, resolves the - feature files and starts no container — which is the appendix's "keep it compiling" in the form + without running it — `pytest tests/tck --collect-only` imports every test module, resolves the + feature files and starts no container — which is the appendix's "keep it compiling" in the form Python has available. ## Adding your own scenarios @@ -138,10 +138,10 @@ calls `scenarios()`, and write step definitions for whatever is new in a `confte ``` tests/ -├── conftest.py # your step definitions -├── test_conformance.py # the fixture and the one call, unchanged -└── extensions/ - └── fractional.feature +├── conftest.py # your step definitions +├── test_conformance.py # the fixture and the one call, unchanged +└── extensions/ + └── fractional.feature ``` ```python @@ -155,7 +155,7 @@ from openfeature.contrib.tools.tck import TckState def fractional_splits(tck_state: TckState) -> None: ... ``` -That is the whole of it — **no registration, no option and no new argument**. pytest collects +That is the whole of it — **no registration, no option and no new argument**. pytest collects `conftest.py` on its own, pytest-bdd resolves steps through the fixture system, and the canonical step vocabulary is in scope in your feature file beside your own steps. `tck_state` is the same per-scenario state the canonical steps use, so your scenario runs against the provider the suite @@ -175,14 +175,14 @@ configuring anything. There used to be a second call, `features_path()`, which returned the canonical set on its own. It is **gone**. The two differed by one character at the call site and the shorter one silently dropped the extensions directory, so reaching for it produced a green run over fewer scenarios than the -adopter believed had run — which is the worst failure mode available to a conformance suite, because +adopter believed had run — which is the worst failure mode available to a conformance suite, because nothing is there to notice. `canonical_root()` is the supported way to reach the packaged directory for anything that is not "the scenarios to run". ### Your scenarios cannot stand in for ours Every feature file carries a uri, and it is how a canonical scenario is told from an adopter's: -canonical files are the ones under the `gherkin/` prefix and yours are under `extensions/` — the +canonical files are the ones under the `gherkin/` prefix and yours are under `extensions/` — the prefix Go and JavaScript mount theirs under too, so a consumer holding conformance reports from several languages applies one rule. Neither prefix is this package's to choose: Appendix F identifies a canonical feature by its path *relative to the specification's asset directory*, which @@ -196,7 +196,7 @@ called it, and `extensions.py` reports two cases that derivation cannot rule out This is not hypothetical. Java's suite found that a same-named feature file in a second classpath root *replaced* the canonical one, and the run went green having asked the adopter's questions -instead of the specification's — the worst outcome available to a conformance suite. The Python +instead of the specification's — the worst outcome available to a conformance suite. The Python route to the same place is narrower and just as quiet: pytest-bdd names a feature file by its parent directory joined to its own name, so `extensions/gherkin/errors.feature` arrives under the uri the canonical `errors.feature` already occupies. @@ -207,7 +207,7 @@ Not every provider implements every optional part of the contract. Each scenario optional part carries a Gherkin tag, pytest-bdd turns that tag into a pytest marker, and a provider declares what it supports. -**A scenario whose capability was not declared is reported as skipped, with the reason — never as +**A scenario whose capability was not declared is reported as skipped, with the reason — never as passed.** A conformance suite that quietly goes green on scenarios it did not run is worse than no suite at all, so `pytest.skip` carries the reason into the report: @@ -227,15 +227,15 @@ SKIPPED provider does not declare capability @stale. | `Capability.DISABLED_FLAGS` | `@disabled-flags` | resolves a flag disabled in the management system to the code default | | `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | | `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | -| `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | +| `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | | `Capability.REINITIALIZATION` | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) permits rather than requires | | `Capability.TARGETING` | `@targeting` | resolves a flag differently for a matching evaluation context | | `Capability.STANDARD_REASONS` | `@standard-reasons` | reports the standard resolution reasons, with the meanings [Appendix F][appendix-f] gives them | -| `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | +| `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | `@lifecycle` and `@events` are deliberately separate, and the split matters in both directions. An SDK dispatches `PROVIDER_READY` around `initialize` for *any* provider, so a provider declaring only -`@events` passes the readiness scenario without demonstrating anything — a `NoOpProvider` passes it +`@events` passes the readiness scenario without demonstrating anything — a `NoOpProvider` passes it identically. Meanwhile a stateless provider has a real initialisation to verify but no event stream of its own to declare `@events` for, and gating on `@events` shut it out of a scenario it should be held to. @@ -249,18 +249,18 @@ again is exercising a choice the specification offers it, so withholding this ca `KnownDeviation` entry. The scenario was untagged until spec revision `fc99d5ac`, on the reading that reverting to the -uninitialized state is observable as exactly one thing — being initialisable again. That inference +uninitialized state is observable as exactly one thing — being initialisable again. That inference does not hold, and asserting it unconditionally reported a permitted choice as a conformance failure. A false failure is the mirror image of a vacuous pass, and this suite cares about both. Reverting the -state is not separately observable either — a provider that reverts but refuses reuse presents -identically to one that did neither — so the gated reuse scenario is the only assertion the +state is not separately observable either — a provider that reverts but refuses reuse presents +identically to one that did neither — so the gated reuse scenario is the only assertion the requirement admits. It is worth keeping for the providers that do offer reuse, because releasing the client on shutdown while leaving an initialised flag set behind is easy to write and leaves the provider evaluating against a closed connection rather than failing outright. One practical note, because it is easy to get wrong: `@reinitialization` **narrows** `@lifecycle` rather than standing beside it. The scenario lives in `lifecycle.feature`, which carries `@lifecycle` -at the feature level, so the scenario inherits it and carries both tags — and the gate skips a +at the feature level, so the scenario inherits it and carries both tags — and the gate skips a scenario when *any* capability gating it is undeclared. Reuse is therefore exercised only by an adoption declaring `Capability.LIFECYCLE` **and** `Capability.REINITIALIZATION`; declaring the latter alone leaves the scenario skipped on `@lifecycle` and the declaration unverified. So a provider that @@ -280,13 +280,13 @@ live in one gated Scenario Outline of eight rows; the value assertions stay unta makes the value a `MUST`. `@targeting` was **reserved and undeclarable** until the same revision, on the reading that targeting -is backend evaluation logic and out of scope. The scope argument still holds — its three scenarios do -not test how a backend evaluates a rule — but the conclusion did not: they exist to show the context +is backend evaluation logic and out of scope. The scope argument still holds — its three scenarios do +not test how a backend evaluates a rule — but the conclusion did not: they exist to show the context reached the backend at all, which is a property of the provider and of nothing else. `targeting-key-flag` is the one flag in the canonical set with a rule, specified by behaviour rather than syntax (resolve `hit` when the targeting key is exactly `5c3d8535-f81a-4478-a6d3-afaa4d51199e`), and a matching context resolving to a different value is what catches a provider that drops the -context — no echo endpoint on the control API required. The three scenarios are the matching context, +context — no echo endpoint on the control API required. The three scenarios are the matching context, the non-matching one and no context at all; the second and third are not padding, since a provider that always returned the targeted value would pass the first and one that refuses to evaluate a rule with no targeting key is caught by the third. @@ -294,20 +294,20 @@ with no targeting key is caught by the third. `@disabled-flags` is gated because it needs two things and only one of them comes for free. The caller's default value is held by the provider, which always has it. What the provider also needs is a **signal** that the flag was disabled, told apart from an ordinary resolution and from a missing -flag — and that belongs to the backend and its protocol. One with no disabled state, or one that +flag — and that belongs to the backend and its protocol. One with no disabled state, or one that answers `FLAG_NOT_FOUND` for a disabled flag, gives the provider nothing to act on. -[Appendix F][appendix-f] draws the line elsewhere — a provider whose backend decides, *"such as one -speaking OFREP, cannot: the server never sees the caller's default, so it has no way to return it"* — +[Appendix F][appendix-f] draws the line elsewhere — a provider whose backend decides, *"such as one +speaking OFREP, cannot: the server never sees the caller's default, so it has no way to return it"* — and what this suite measured does not bear that out. flagd's RPC resolver is a remote evaluator by exactly that description and satisfies the capability: the server answers reason `DISABLED` with no variant and no value, and the resolver substitutes the caller's default locally on that signal. flagd's OFREP endpoint answers the same flag with `{"reason": "DISABLED"}` and no `value` and no -`variant` — the same signal in another envelope — and the Python OFREP provider already falls back to +`variant` — the same signal in another envelope — and the Python OFREP provider already falls back to the caller's default for the absent value. It fails these scenarios for a reason unrelated to architecture, which [its own suite](../../providers/openfeature-provider-ofrep/tests/tck/test_ofrep_conformance.py) records. The discrepancy belongs upstream rather than papered over here; what it changes locally is -only what a withheld declaration may be read as — not necessarily an impossibility, so read the +only what a withheld declaration may be read as — not necessarily an impossibility, so read the adoption's own note for which it was. Withholding still needs no `KnownDeviation`, for the reason every gated capability does: a deviation records a gap in behaviour the provider is *required* to have, and this one is optional. @@ -320,9 +320,9 @@ does for `@numeric-coercion`, and gates it. Since spec revision `009afe06` the c four `disabled-*` flags mirroring `boolean-flag`, `string-flag`, `integer-flag` and `float-flag` exactly, differing only in `state`, and one Scenario Outline of four rows asserts that each resolves to the caller's default. Each row's default differs from the flag's configured value, so a provider -that ignores the state is caught on the value alone — 2.2.3, a `MUST`. The rows assert neither the -reason, which would rest on 2.2.5's `SHOULD` and its "some other string" — it is pinned in -`gherkin/reason.feature` instead, for a provider that opts into the standard meanings — nor the +that ignores the state is caught on the value alone — 2.2.3, a `MUST`. The rows assert neither the +reason, which would rest on 2.2.5's `SHOULD` and its "some other string" — it is pinned in +`gherkin/reason.feature` instead, for a provider that opts into the standard meanings — nor the variant, since a disabled flag has resolved none: `@disabled-flags` and `@variants` deliberately do not compose. @@ -338,14 +338,14 @@ that silently falls back is already caught by the value. So the reasons live in `gherkin/reason.feature`, gated as a whole. Declaring the capability is a provider saying *"I use the standard vocabulary with the standard meanings"*, and that file is what -checks the claim — `STATIC` for a rule-less flag, `TARGETING_MATCH` for a matched rule, `DEFAULT` for +checks the claim — `STATIC` for a rule-less flag, `TARGETING_MATCH` for a matched rule, `DEFAULT` for an unmatched one, `DISABLED` for a disabled flag, `ERROR` beside an error code. A provider that does not declare it **loses nothing**: its values, variants and error codes are asserted everywhere else, -on `MUST` requirements. What the declaration adds is something a report's reader can act on — anyone +on `MUST` requirements. What the declaration adds is something a report's reader can act on — anyone building telemetry, dashboards or debugging on `reason` can see that the vocabulary was verified rather than assumed. `STATIC` for the rule-less rows is the call worth flagging: `types.md` types `DEFAULT` as *"no dynamic evaluation occurred **or** dynamic evaluation yielded no result"*, so a -provider answering `DEFAULT` there is not defective — it does not use the standard meanings, and +provider answering `DEFAULT` there is not defective — it does not use the standard meanings, and should not declare the tag. **Tags compose, and here that is load-bearing.** `TARGETING_MATCH` cannot be observed without @@ -355,61 +355,98 @@ of the file's scenarios also carry `@targeting` and one also carries `@disabled- other three with their reason. Untagged scenarios are mandatory and always run. `capabilities` defaults to every *declarable* -capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from +capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from the default, run the suite, and remove only what your provider genuinely cannot do. Leaving a capability out is the only way to withhold it, and one skip carrying its reason is the whole mechanism: the scenario's tags say what was asked, the declaration says whether it was -claimed, and the skip says why it was not. A capability that cannot hold in a language *at all* — -`@numeric-coercion` where the language has a single numeric type, `@large-integers` on a 32-bit -accessor — is a property of the SDK rather than of the provider, and -[Appendix F][appendix-f] records it once instead of every report restating it. +claimed, and the skip says why it was not. + +### A capability this SDK cannot express + +Some capabilities cannot hold in a language *at all* — `@numeric-coercion` where the language has a +single numeric type and "a float requested as an integer" does not name two different requests, +`@large-integers` where the integer accessor is a 32-bit `Integer`. That is a property of the SDK +rather than of the provider, so [Appendix F][appendix-f] makes it the implementation's job: +`INEXPRESSIBLE_CAPABILITIES` lists them, `TckConfig` refuses to let you declare one, and the error +names the property of the SDK that puts the question out of reach. You are not expected to know this +about your language, and three suites each remembering it separately is three chances to put a claim +in a report that no scenario could have verified. + +**`INEXPRESSIBLE_CAPABILITIES` is empty in Python, and that was measured rather than assumed.** +`int` is arbitrary-precision, and `get_integer_details` and `get_float_details` are separate +accessors reaching separate provider methods and type-checked against `int` and `float` separately — +so all four questions the two tags ask can be put, and all four were asked and answered. Both are +ordinary declarable capabilities here. + +A provider that gets one of them *wrong* is a different thing and does not belong here. Measured by +declaring `@numeric-coercion` in both flagd suites and running it: flagd's in-process resolver +refuses `0.5` as an integer and widens `10` to a float, and its RPC resolver widens `10` and +silently narrows `0.5` to `0`, which is the one thing the lossy scenario forbids. Two different +answers to the same three questions, from two resolvers of one provider — which is what a language +that *could not ask* them would make impossible. Both are defects in an implementation, withholding +the tag is the honest report for each, and neither has anything to do with Python. (The third +scenario fails on both for a third reason again: flagd-testbed does not seed `integral-float-flag` +at all.) + +**A reservation and an inexpressibility are not the same refusal**, and the messages and the skip +reasons deliberately differ: + +| | reserved (`@caching`) | inexpressible | +|---|---|---| +| Why | no scenario anywhere carries the tag | the scenarios exist and this SDK cannot ask them | +| Scope | every language | one language | +| Lifetime | expires when the specification adds scenarios | permanent, until the SDK changes | +| The skip says | the capability has no scenarios yet | no provider in this language can be asked | + +Anyone reading a report has to be able to tell *"this provider declined"* from *"no provider in this +language can be asked"*, because only the first says anything about the provider. A reserved capability is documented so the vocabulary has a place for it once scenarios exist, and until then it **must not be declared**. Nothing carries the tag, so declaring it cannot be verified, cannot produce a skip, and tells anyone reading the declaration only that something was claimed and nothing examined. `TckConfig` raises if you name one in `capabilities`, and -`DECLARABLE_CAPABILITIES` excludes them — which is the case that matters, because "every capability +`DECLARABLE_CAPABILITIES` excludes them — which is the case that matters, because "every capability except X" is how a reserved tag gets declared by accident rather than by decision. One implementation's published conformance report asserts `@targeting` and `@caching` for exactly that reason, back when both were reserved. `@caching` is the only reserved tag left. Leaving one reserved once it *has* scenarios would be the -mirror of the mistake the set exists to prevent — a capability that can be verified, refused the -chance — so `@targeting` moved out of it the moment the specification gave it three. +mirror of the mistake the set exists to prevent — a capability that can be verified, refused the +chance — so `@targeting` moved out of it the moment the specification gave it three. **A scenario carrying a reserved tag fails the run.** `TckConfig` refuses to let anyone declare a -reserved capability, so the gate skips every scenario carrying one — for a capability nobody is +reserved capability, so the gate skips every scenario carrying one — for a capability nobody is permitted to claim, which leaves a gap in the report that the provider may not have. Appendix F calls that the unclaimable capability, and nothing else notices it: the run is green and the report is well-formed. So the plugin refuses to continue, naming the tags. Two mistakes end there and the message names both remedies. Either the tag arrived with the canonical feature files, because the specification wrote the scenarios the reservation was held open for and -this package has not followed — take the tag out of `RESERVED_CAPABILITIES` and decide, per adoption, +this package has not followed — take the tag out of `RESERVED_CAPABILITIES` and decide, per adoption, whether to declare it. Or it arrived from a feature file of your own under `extensions/`, in which case pick a tag of your own: a reserved tag gates nothing and can never be declared, so a scenario carrying one can never run. The two halves are read differently, and that is deliberate. The canonical tags come from the packaged files rather than from the collected run, so a `-k` or `--deselect` cannot narrow a run past -the specification's half; your extensions have no such source — the directory is found from your test -module — so those tags come from what was collected. Both are the parsed Gherkin tags, never the file +the specification's half; your extensions have no such source — the directory is found from your test +module — so those tags come from what was collected. Both are the parsed Gherkin tags, never the file text: `gherkin/events.feature` names `@caching` in a `#` comment saying where those scenarios will go once they exist, and a text scan would fail every adoption over a sentence. `@numeric-coercion` deserves a note, because it is the one capability here that **the specification -does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of +does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of unspecified type or size", and languages **may** differentiate between integers and floats "as idioms -dictate" — so no requirement says what a provider must do when a value does not fit the accessor it +dictate" — so no requirement says what a provider must do when a value does not fit the accessor it was asked through. That gap is [open-feature/spec#430](https://github.com/open-feature/spec/issues/430). The rule this tag is tested against is therefore **borrowed, not normative**: lossless coercion is -permitted, lossy coercion must fail — `10.0` requested as an integer must succeed, `0.5` must not. +permitted, lossy coercion must fail — `10.0` requested as an integer must succeed, `0.5` must not. It comes from flagd's [numeric coercion ADR](https://github.com/open-feature/flagd/blob/main/docs/architecture-decisions/numeric-coercion.md), -which is scoped to flagd's own implementations, and the tag carries that name — it was -`@strict-numeric-typing` — because two vocabularies for one observable property is worse than one +which is scoped to flagd's own implementations, and the tag carries that name — it was +`@strict-numeric-typing` — because two vocabularies for one observable property is worse than one borrowed name. **A provider that behaves differently is not violating the specification**, so withholding this capability may be a deliberate choice as readily as a defect. @@ -419,15 +456,15 @@ integer is `10`; `integer-flag` (`10`) requested as a float is `10.0`. Rejecting way to pass the first, and the other two are what stop it. The width of the integer accessor is the related property, and it is a capability of its own because -it belongs to the SDK rather than to the provider. Every language can ask for 2^31 − 1, so that -precision scenario is untagged; only the one asking for 2^53 − 1 carries `@large-integers`. A Python -`int` is unbounded, so a Python provider declares it unless something of its own — a 32-bit field in -its wire format, a float on the way through — narrows the value. +it belongs to the SDK rather than to the provider. Every language can ask for 2^31 − 1, so that +precision scenario is untagged; only the one asking for 2^53 − 1 carries `@large-integers`. A Python +`int` is unbounded, so a Python provider declares it unless something of its own — a 32-bit field in +its wire format, a float on the way through — narrows the value. ### Steps that reach the provider directly Everything the suite asks of a provider goes through an OpenFeature client, as an application's -would — except three steps. `the provider is shut down` and `the provider is initialized again` call +would — except three steps. `the provider is shut down` and `the provider is initialized again` call the provider's own `shutdown()` and `initialize()` on the registered instance, and `the provider metadata name should not be empty` asks it for `get_metadata()`. Going through the SDK would test the registry's bookkeeping as much as the provider, and Appendix B already does that; it @@ -436,7 +473,7 @@ per registration. The registry is not told. The client keeps pointing at the same instance, so an evaluation after re-initialising reaches the very object that was shut down and brought back. When the scenario ends, -the SDK shuts the provider down once more on its own — requirement 2.5.3 makes that second call +the SDK shuts the provider down once more on its own — requirement 2.5.3 makes that second call harmless, and the suite relies on it. A direct call that outlasts `TckConfig.ready_timeout` is given up on and fails its scenario with a message rather than hanging the session. @@ -489,7 +526,7 @@ field exists to prevent. Where the specification permits the choice, withholding definitions never talk to a backend directly, which is why the same Gherkin runs unchanged against a containerised backend and against a provider manipulated in-process. -**If your provider talks to a backend, drive it over the HTTP control API** — the document is +**If your provider talks to a backend, drive it over the HTTP control API** — the document is available as `control_api_spec()`, and `HttpControl` is the client for it. That API is the normative contract for those providers, and it is what makes a conformance claim portable: another language's TCK drives the same endpoints against the same stack and must get the same answers. @@ -503,26 +540,26 @@ one yourself when you use the Compose harness below: it is handed to you as `tck already awaited ready. **A control must say which path it drove the backend through.** `control_api` is a required member of -`BackendControl`, typed `ControlApi` — `Literal["http", "in-process"]` — with no default and no +`BackendControl`, typed `ControlApi` — `Literal["http", "in-process"]` — with no default and no inference from the control's concrete type. `HttpControl` answers `"http"`, `InProcessControl` answers `"in-process"`, and a custom control states its own. It is the one fact that decides what everything else in a report is worth: the same scenarios passing over the control API and passing through in-process manipulation of a provider that *does* have a backend are not the same claim, and this is the only field that separates them. Nothing outside a control can tell the two apart, and -every run is one or the other — so an absent value would not be "no claim made" but an unfalsifiable +every run is one or the other — so an absent value would not be "no claim made" but an unfalsifiable one. The type is closed, so `"HTTP"` or `"grpc"` is a type error here rather than a conformance report that fails schema validation somewhere else. ### The container stack -The suite starts it. An adopter used to write the container wrapper — and every adopter wrote the +The suite starts it. An adopter used to write the container wrapper — and every adopter wrote the same one, which is why the flagd adoption alone carried a 122-line `conftest.py` and a 170-line `suite.py` of it. `ComposeBackend` is the whole declaration: | field | required | default | meaning | | --- | --- | --- | --- | -| `compose_file` | yes | — | path to the Compose file, resolved relative to the package directory | -| `backend_ports` | yes | — | container-internal ports the **provider** connects to. The control port is exposed automatically and must not be listed here | +| `compose_file` | yes | — | path to the Compose file, resolved relative to the package directory | +| `backend_ports` | yes | — | container-internal ports the **provider** connects to. The control port is exposed automatically and must not be listed here | | `backend_service` | no | `"backend"` | the Compose service hosting both the control API and the backend | | `control_port` | no | `8080` | container-internal port of the control API | | `additional_ports` | no | `{}` | extra service to ports, for a stack with more than one service. Resolved through the endpoint by service name | @@ -534,10 +571,10 @@ them writes one Compose file and two declarations against it. `tck_backend` is a session-scoped fixture this package's plugin supplies, and it yields two things: -- `tck_backend.control` — the `HttpControl`, already awaited ready. Hand it to `TckConfig.control`. +- `tck_backend.control` — the `HttpControl`, already awaited ready. Hand it to `TckConfig.control`. One per stack: it remembers whether a disconnect left the backend down, so two suites driving the same backend must share it. -- `tck_backend.endpoint` — `host`, `port(internal)` and `port(internal, service=...)`. This is a +- `tck_backend.endpoint` — `host`, `port(internal)` and `port(internal, service=...)`. This is a **factory argument, not a field**: the mapped ports do not exist until the stack is up, which is why `TckConfig.new_provider` is a factory called once per scenario. @@ -551,18 +588,18 @@ Startup is a real readiness check rather than a pause: the stack comes up with `docker compose up --wait`, then every declared port is waited on until it accepts a connection, then `HttpControl.await_ready()` probes `GET /healthz` until the control API answers. There is deliberately **no settle after a control call**. Java had a fixed 50ms one, and `control-api.yaml` -now states what makes it the wrong instrument: every state-changing endpoint — `/start`, `/change`, -`/reset` — must not return until the new state is actually being served, so a delay here covers a +now states what makes it the wrong instrument: every state-changing endpoint — `/start`, `/change`, +`/reset` — must not return until the new state is actually being served, so a delay here covers a window the backend is specified to close, and a suite that sleeps instead of holding the API to that promise stops being able to detect when the promise breaks. The delay is also un-tunable, because the window is a property of the backend and not of the harness. -Backends do still break it — flagd-testbed's launchpad returns from `/start` as soon as `/readyz` +Backends do still break it — flagd-testbed's launchpad returns from `/start` as soon as `/readyz` answers, which is roughly 40 ms before the flags are evaluable, and [flagd-testbed#394](https://github.com/open-feature/flagd-testbed/pull/394) is open and unmerged. A provider that blocks in `initialize` absorbs that window; a stateless one lands in it. Where you are stuck with such a backend the wait belongs in **your adoption**, set explicitly and citing the -defect, so it reads as a named workaround for one backend and disappears when the backend is fixed — +defect, so it reads as a named workaround for one backend and disappears when the backend is fixed — see the OFREP adoption's `SettledControl`. It does not belong here, where every future adopter would inherit it without knowing why. @@ -605,7 +642,7 @@ Two of the API's requirements are easy to get wrong: baseline. - **There is no binding for `POST /restart`.** It simulates a *bounded* outage and is `[OPTIONAL]` in `control-api.yaml`, because no shipped scenario reaches it: the disconnect/reconnect scenario is - written as an unbounded outage — "the connection is lost", then "the connection is restored" — + written as an unbounded outage — "the connection is lost", then "the connection is restored" — which is `disconnect()` then `reconnect()`, so the scenario ends the outage when it is ready rather than guessing in advance how long the provider needs to notice one. What would bring the endpoint back is a `@caching` scenario asserting what a stale provider serves *during* an outage, which @@ -618,8 +655,8 @@ control the backend in-process, where flag operations are direct manipulations o state. `InProcessControl` is the reference. This is a narrow allowance and the obvious thing to abuse. **A provider with an external backend -must use the control API.** Reaching into an external backend from inside the test process — a -test-only admin client, a shared database handle, a hook inside the provider — produces a suite that +must use the control API.** Reaching into an external backend from inside the test process — a +test-only admin client, a shared database handle, a hook inside the provider — produces a suite that passes while proving nothing, because the path it exercised is not the path the contract describes. Connection-dependent scenarios have no meaning without a connection, so a backend-less control @@ -639,7 +676,7 @@ Four, all confirmed by running the suite rather than by reading code. error code**, where the specification requires the code default and `TYPE_MISMATCH`. The client type-checks with `isinstance(value, int)`, and `bool` is a subclass of `int` in Python. -This is **Python-specific** — the identical scenario passes in every other language's suite, which +This is **Python-specific** — the identical scenario passes in every other language's suite, which is a fair advertisement for having more than one implementation. Tracked as [open-feature/python-sdk#619](https://github.com/open-feature/python-sdk/issues/619). @@ -653,8 +690,8 @@ emitting `PROVIDER_CONFIGURATION_CHANGED`. Python's copies its mapping in the co exposes nothing to change it. Tracked as [open-feature/python-sdk#620](https://github.com/open-feature/python-sdk/issues/620). -Only half the machinery is missing — `AbstractProvider` already supplies -`emit_provider_configuration_changed` — which is why `ControllableInMemoryProvider` here is a small +Only half the machinery is missing — `AbstractProvider` already supplies +`emit_provider_configuration_changed` — which is why `ControllableInMemoryProvider` here is a small subclass rather than a reimplementation, and why it should port back to the SDK as a method. ### 3. The in-memory provider does not coerce numbers @@ -662,7 +699,7 @@ subclass rather than a reimplementation, and why it should port back to the SDK `integral-float-flag` (`10.0`) requested as an integer returns the code default with `TYPE_MISMATCH`, and `integer-flag` (`10`) requested as a float does the same. The provider hands each variant back untouched and the client's type check is `isinstance`-based, so neither lossless direction happens. -The lossy scenario passes — every float is rejected — which is exactly the shortcut the two lossless +The lossy scenario passes — every float is rejected — which is exactly the shortcut the two lossless scenarios exist to catch. This is not a defect: `@numeric-coercion` is optional, and the specification does not define the @@ -677,7 +714,7 @@ reason `STATIC` whatever the state, and `InMemoryProvider._resolve` looks only f all four `disabled-*` flags are served exactly as their enabled counterparts are. `canonical_flag_set` is not where this stops. `_decode_canonical_flags` reads the canonical file's -`"state": "DISABLED"`, validates it against `InMemoryFlag.State` and passes it through faithfully — +`"state": "DISABLED"`, validates it against `InMemoryFlag.State` and passes it through faithfully — the self-tests pin that it reaches exactly those four flags and no others. The state survives decoding and then has no effect. @@ -686,7 +723,7 @@ Measured before the tag was gated: all four rows failed on the value in both in- declares `@disabled-flags` and the four scenarios are skipped with that reason. Unlike finding 3 this is a field the SDK offers and does not honour, which is closer to a defect than -to a choice — but the capability is optional, so the honest report is still a withheld declaration +to a choice — but the capability is optional, so the honest report is still a withheld declaration rather than a `KnownDeviation`. It is not filed against the SDK yet. ## Where the assets come from @@ -694,7 +731,7 @@ rather than a `KnownDeviation`. It is not filed against the SDK yet. The Gherkin feature files, the canonical flag set and the control-API document are **not owned by this repository**. They are the language-agnostic conformance artifacts defined in [open-feature/spec][spec] under `specification/assets/provider-tck/`, and every language's TCK ships -the same ones — which is the only reason a conformance claim means the same thing in Python as it +the same ones — which is the only reason a conformance claim means the same thing in Python as it does in Java. **Adopting this package needs no submodule.** The assets are copied into the wheel and the sdist at @@ -759,8 +796,8 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | Suite | Subject | Why | | --- | --- | --- | | `test_in_memory_conformance` | the SDK's `InMemoryProvider` | reference adoption for a backend-less provider | -| `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | -| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set is decoded from `canonical-flags.json` — every flag served under the file's own default variant, with the Python type the file wrote, and `state` reaching exactly the four `disabled-*` flags | +| `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | +| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set is decoded from `canonical-flags.json` — every flag served under the file's own default variant, with the Python type the file wrote, and `state` reaching exactly the four `disabled-*` flags | | `test_lifecycle_steps` | the steps that call the provider directly | the in-memory suites skip `@lifecycle`, so the shutdown, re-initialise and metadata steps are driven against a recording provider instead | | `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | | `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity | @@ -768,29 +805,29 @@ This mirrors what `openfeature-flagd-api-testkit` already does for the flagd tes | `test_spec_assets` | where the conformance assets came from | the submodule checkout and the copy are the only thing standing between a run and the wrong questions, and a stale copy is invisible in a pass or a fail | ``` -214 passed, 42 skipped, 2 xfailed +221 passed, 42 skipped, 2 xfailed ``` No Docker and no network beyond loopback. The conformance suites take under a second; `test_extensions` takes most of the rest, because the properties it checks are properties of a whole pytest session and it runs a generated adoption in a subprocess to check them. -Neither in-memory suite declares `@lifecycle`, so the six lifecycle scenarios — three about -initialisation, three about shutdown — are skipped in both. That is the point: with no backend to -reach, the initialisation ones would pass without testing anything — which is what they did while the +Neither in-memory suite declares `@lifecycle`, so the six lifecycle scenarios — three about +initialisation, three about shutdown — are skipped in both. That is the point: with no backend to +reach, the initialisation ones would pass without testing anything — which is what they did while the feature was gated on `@events`. Neither declares `@numeric-coercion` either, for the reason in finding 3, so its three scenarios are skipped too. Neither declares `@targeting`: both resolve the same decoded flag set, and `canonical_flag_set` deliberately ignores `targeting-key-flag`'s rule rather than becoming a second implementation of somebody else's evaluator, so those three scenarios -are skipped as well. Neither declares `@disabled-flags` either, for the reason in finding 4 — the -state reaches the flag set and the SDK's provider never reads it — so its four rows are skipped in +are skipped as well. Neither declares `@disabled-flags` either, for the reason in finding 4 — the +state reaches the flag set and the SDK's provider never reads it — so its four rows are skipped in both. Both declare `@variants`, since an in-memory flag set is keyed by variant name. Both declare `@standard-reasons`, and it was measured before it was declared: `InMemoryFlag.resolve` reports `Reason.STATIC` for every flag in the decoded set, and a missing flag and a type mismatch both arrive with reason `ERROR` beside their error code, so the four rule-less rows and the two error scenarios pass in each suite. The remaining three scenarios in `reason.feature` compose the tag with -`@targeting` and `@disabled-flags`, neither of which is declared, so they are skipped in both — which +`@targeting` and `@disabled-flags`, neither of which is declared, so they are skipped in both — which is the composition working rather than a gap, since a reason cannot be observed without the behaviour that produces it. @@ -798,7 +835,7 @@ that produces it. - **Evaluation context passthrough is verified only for the targeting key.** `targeting-key-flag` resolves differently for a matching context, so the `@targeting` scenarios catch a provider that - drops the context — no echo operation needed for that. What is still unverified is that the + drops the context — no echo operation needed for that. What is still unverified is that the *whole* context arrives intact: a provider that forwards the targeting key and silently discards every other attribute passes. That needs either an echo operation on the control API or a second canonical flag whose rule keys on a custom attribute. diff --git a/tools/openfeature-tck/spec b/tools/openfeature-tck/spec index c342461aa..89b1519a0 160000 --- a/tools/openfeature-tck/spec +++ b/tools/openfeature-tck/spec @@ -1 +1 @@ -Subproject commit c342461aa95df9e3b46320dbae65e88e5e8b815a +Subproject commit 89b1519a08d81c46ba47fc2a54c44d40fdee845d diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py index 7572ae67a..d0b7f91fe 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py @@ -81,7 +81,12 @@ def tck_config(): import importlib.resources -from .capability import DECLARABLE_CAPABILITIES, RESERVED_CAPABILITIES, Capability +from .capability import ( + DECLARABLE_CAPABILITIES, + INEXPRESSIBLE_CAPABILITIES, + RESERVED_CAPABILITIES, + Capability, +) from .compose import ( DEFAULT_BACKEND_SERVICE, DEFAULT_CONTROL_PORT, @@ -125,6 +130,7 @@ def tck_config(): "DEFAULT_CONTROL_PORT", "DEFAULT_STARTUP_TIMEOUT", "EXTENSIONS_DIRECTORY", + "INEXPRESSIBLE_CAPABILITIES", "RESERVED_CAPABILITIES", "BackendControl", "BackendEndpoint", diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py index 52bc45d0f..ec5629e56 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py @@ -3,7 +3,9 @@ from __future__ import annotations import typing +from collections.abc import Mapping from enum import Enum +from types import MappingProxyType __all__ = ["Capability"] @@ -24,6 +26,12 @@ class Capability(str, Enum): run is worse than no suite at all. Scenarios with no capability tag are mandatory and always run. + + Two kinds of capability are refused rather than declared, and they are + refused for different reasons and with different messages: + :data:`RESERVED_CAPABILITIES`, which no scenario anywhere carries yet, and + :data:`INEXPRESSIBLE_CAPABILITIES`, whose question this language's SDK cannot + put at all. :data:`DECLARABLE_CAPABILITIES` is what is left. """ LIFECYCLE = "lifecycle" @@ -199,8 +207,13 @@ class Capability(str, Enum): The SDK's own ``InMemoryProvider`` cannot declare this: it hands values back untouched and the client's type check is ``isinstance``-based, so ``10.0`` requested as an integer is a ``TYPE_MISMATCH`` rather than ``10``. - The width of the integer accessor is a separate property, and a separate - capability: :attr:`LARGE_INTEGERS`. + That is the provider declining to coerce, not the language refusing to ask: + a provider that does coerce returns an ``int`` and the same check passes it. + In a language with one numeric type the question could not be put at all, + which is why Appendix F names this as inexpressible there and why + :data:`INEXPRESSIBLE_CAPABILITIES` is empty here. The width of the integer + accessor is a separate property, and a separate capability: + :attr:`LARGE_INTEGERS`. """ LARGE_INTEGERS = "large-integers" @@ -214,7 +227,10 @@ class Capability(str, Enum): Python's ``int`` is unbounded, so a Python provider declares it unless something of its own -- a 32-bit field in its wire format, a float on the - way through -- narrows the value. Nothing above 2^53 - 1 is asked for: + way through -- narrows the value. Which makes this one of the two + capabilities Appendix F names as inexpressible somewhere and **not** here: + :data:`INEXPRESSIBLE_CAPABILITIES` is empty in Python, and says on what + measurement. Nothing above 2^53 - 1 is asked for: JavaScript cannot represent it, and what a provider owes a value that does not fit the requested accessor is the open question in `open-feature/spec#430 `_. @@ -394,6 +410,25 @@ def reserved(self) -> bool: """Whether this capability exists in the vocabulary but gates no scenario.""" return self in RESERVED_CAPABILITIES + @property + def inexpressible(self) -> bool: + """Whether this SDK cannot put the question this capability's scenarios ask. + + Distinct from :attr:`reserved` in every respect except that both end in a + refusal. See :data:`INEXPRESSIBLE_CAPABILITIES`. + """ + return self in INEXPRESSIBLE_CAPABILITIES + + @property + def inexpressible_reason(self) -> str | None: + """Which property of this SDK puts the question out of reach, or ``None``. + + The property, not the rule: a message that only says "this cannot be + declared" leaves the adopter to discover why, and the why is the part + they could not have been expected to know. + """ + return INEXPRESSIBLE_CAPABILITIES.get(self) + def __str__(self) -> str: return self.tag @@ -418,10 +453,60 @@ def __str__(self) -> str: capability that *can* be verified and is refused the chance. """ +INEXPRESSIBLE_CAPABILITIES: Mapping[Capability, str] = MappingProxyType({}) +"""Capabilities this language's SDK cannot put the question for, and why. + +**Empty in Python, and that is a measurement rather than an omission.** The two +that exist anywhere are :attr:`Capability.LARGE_INTEGERS`, inexpressible where +the integer accessor is a 32-bit ``Integer``, and +:attr:`Capability.NUMERIC_COERCION`, inexpressible where the language has a +single numeric type and "a float requested as an integer" does not name two +different requests. Python has neither property: ``int`` is arbitrary-precision, +and ``get_integer_details`` and ``get_float_details`` are separate accessors +reaching separate provider methods, type-checked against ``int`` and ``float`` +separately. Both were checked by asking all four questions through the SDK +rather than by reading its source, and every one of them was answered. + +So this mapping carries no entries, and the machinery around it carries no load +here. It exists anyway because the rule is Appendix F's rather than this +package's, because the next capability may hit it, and because the cost of the +two is not symmetric: an unused mechanism is a few lines nobody reads, while a +missing one is discovered by an adopter publishing a claim no scenario could +have examined. + +**Not the same thing as a reservation, and the difference is what the two +messages have to carry.** A reserved capability is global and temporary -- no +scenario anywhere carries the tag, and the reservation expires the moment the +specification writes one. An inexpressible capability is one language's and +permanent: the scenarios exist, other languages run them and pass them, and +nothing changes until the SDK does. A reader seeing a capability missing from a +report has to be able to tell *"this provider declined"* from *"no provider in +this language can be asked"*, because only the first says anything about the +provider. Hence a mapping rather than a set: the value is the property of the +SDK that puts the question out of reach, and it is the half of the message an +adopter could not have worked out for themselves. + +A capability belongs here only when **no** provider in this language could ever +satisfy it. A provider that gets the answer wrong is a different thing entirely +and belongs nowhere near this mapping. flagd's two Python resolvers answer the +three ``@numeric-coercion`` scenarios differently from each other: in-process +refuses ``0.5`` as an integer and widens ``10`` to a float, while RPC widens +``10`` and silently narrows ``0.5`` to ``0``. Each is a defect in an +implementation, recorded where that adoption records its defects, and +withholding the tag is the honest report for both. Listing it here would say the +question cannot be asked -- and two resolvers of one provider giving different +answers to it is the proof that it can. + +Never overlaps :data:`RESERVED_CAPABILITIES`: a tag no scenario carries is +reserved, whatever any SDK could express about it. +""" + DECLARABLE_CAPABILITIES: frozenset[Capability] = ( - frozenset(Capability) - RESERVED_CAPABILITIES + frozenset(Capability) + - RESERVED_CAPABILITIES + - frozenset(INEXPRESSIBLE_CAPABILITIES) ) -"""Every capability an adoption may declare: the vocabulary minus the reserved tags. +"""Every capability an adoption may declare: the vocabulary minus what is refused. A reasonable starting point for a new adoption: declare everything, run the suite, and remove only what the provider genuinely cannot do. Narrowing from this @@ -434,6 +519,12 @@ def __str__(self) -> str: way past, which is how one implementation came to report ``@targeting`` and ``@caching`` as declared without anyone deciding to claim them -- back when both were reserved. + +It excludes :data:`INEXPRESSIBLE_CAPABILITIES` for the same reason and one more: +that set is empty in Python, so a default spanning the whole enum would look +correct here forever and be wrong the day an entry is added, in the one language +where it was added. Derived rather than listed, so it cannot be the thing that is +out of date. """ _BY_MARKER: dict[str, Capability] = {c.value: c for c in Capability} diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py index 13ba53f91..19e8f67de 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py @@ -89,7 +89,11 @@ class KnownDeviation: to no capability. A reserved capability is refused: no scenario carries the tag, so there is - nothing to deviate from. See :data:`~.capability.RESERVED_CAPABILITIES`. + nothing to deviate from. See :data:`~.capability.RESERVED_CAPABILITIES`. So + is one this SDK cannot express, for the opposite reason -- the scenarios + exist and no provider here can attempt them, so the gap is the language's + and not this provider's. See + :data:`~.capability.INEXPRESSIBLE_CAPABILITIES`. """ @classmethod @@ -214,11 +218,14 @@ class TckConfig: Naming a reserved capability here is rejected at construction rather than passed into a report. See :data:`~.capability.RESERVED_CAPABILITIES`. - A capability that cannot hold in a language at all -- ``@numeric-coercion`` - where the language has a single numeric type, ``@large-integers`` on a - 32-bit accessor -- is a property of the SDK rather than of the provider, and - Appendix F records it once rather than every report restating it. Here it is - simply left undeclared, and the skip carries the reason. + So is one this language's SDK cannot put the question for at all -- + ``@numeric-coercion`` where the language has a single numeric type, + ``@large-integers`` on a 32-bit accessor. That is a property of the SDK + rather than of the provider, so it is refused here rather than left for + every adopter to know and remember, and the error names the property. The + two refusals are deliberately not the same message, and the scenarios they + skip do not carry the same reason: see + :data:`~.capability.INEXPRESSIBLE_CAPABILITIES`, which is empty in Python. """ known_deviations: Sequence[KnownDeviation] = () @@ -283,6 +290,7 @@ def __post_init__(self) -> None: object.__setattr__(self, "known_deviations", tuple(self.known_deviations)) problems.extend(reserved_problems(self.capabilities)) + problems.extend(inexpressible_problems(self.capabilities)) problems.extend(deviation_problems(self.known_deviations)) if ( @@ -353,6 +361,51 @@ def reserved_problems(declared: Iterable[Capability]) -> list[str]: ] +def inexpressible_problems(declared: Iterable[Capability]) -> list[str]: + """Refuse a capability this language's SDK cannot put the question for. + + Refused here rather than left to adopters, because leaving it to adopters + means every adopter in the language has to know a fact about their language + and remember to act on it. Three suites in one implementation each left the + same capability undeclared with its own comment restating the same property + of the language: three places to get right, every one of them re-paid by the + next adoption, and a single wrong one puts a claim in a report that no + scenario could have verified. Appendix F makes this the implementation's job + for exactly that reason. + + **The message names the property of the SDK, not the rule.** An adopter who + reaches this has done nothing wrong -- they declared a capability their + provider may well have -- so the error has to tell them something they could + not have known, and "the specification says you may not" is not it. + + Separate from :func:`reserved_problems` on purpose, and it stays separate + even though both end in the same refusal. A reserved capability is global and + temporary: nothing anywhere carries the tag, and the reservation expires when + the specification writes a scenario. An inexpressible one is this language's + and permanent: the scenarios exist and other languages pass them. Collapsing + them into one predicate would make the two indistinguishable at the only + moment anybody is looking. + """ + refused = [ + capability + for capability in declared + if isinstance(capability, Capability) and capability.inexpressible + ] + if not refused: + return [] + return [ + f"{capability.tag} cannot be declared in this language: {reason}. No " + f"provider in this SDK can be asked the question its scenarios put, so a " + f"declaration could not be verified either way, and its absence from a " + f"report says nothing about your provider. Its scenarios are skipped with " + f"that reason. This is not a reservation -- the scenarios exist and other " + f"languages run them -- and there is nothing for you to fix; it changes " + f"when the SDK does" + for capability in sorted(refused, key=lambda c: c.tag) + if (reason := capability.inexpressible_reason) is not None + ] + + def deviation_problems(deviations: Sequence[KnownDeviation]) -> list[str]: """Refuse a deviation that says nothing a consumer can use. @@ -398,6 +451,15 @@ def deviation_problems(deviations: Sequence[KnownDeviation]) -> list[str]: f"could show the gap. Remove it, or name the capability whose " f"scenarios the gap actually affects" ) + elif capability.inexpressible: + problems.append( + f"known_deviations[{index}] names {capability.tag}, which cannot " + f"be expressed in this language: {capability.inexpressible_reason}. " + f"A deviation says this provider fails something it is required to " + f"do, and no provider in this SDK can attempt these scenarios at " + f"all -- so the entry would attribute to your provider a gap that " + f"belongs to the language. The skip already carries that reason" + ) return problems diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py index 1005fa87a..6541ced4a 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/plugin.py @@ -20,6 +20,7 @@ from __future__ import annotations import typing +from collections.abc import Iterable from pathlib import Path import pytest @@ -212,6 +213,16 @@ def tck_state(tck_config: TckConfig) -> typing.Iterator[TckState]: @pytest.fixture(autouse=True) def _tck_capability_gate(request: pytest.FixtureRequest) -> None: + """Autouse wrapper around :func:`capability_gate`. + + A one-line fixture over a plain function, so the decision it makes can be + put under test without reaching inside a fixture object for the callable + pytest wrapped -- which is private, and has moved between pytest versions. + """ + capability_gate(request) + + +def capability_gate(request: pytest.FixtureRequest) -> None: """Skip a scenario whose capability the provider did not declare. ``pytest.skip`` here reports the scenario as skipped **with the reason**, @@ -226,6 +237,17 @@ def _tck_capability_gate(request: pytest.FixtureRequest) -> None: Checking markers first also means the gate costs nothing, and instantiates nothing, for tests that are not TCK scenarios. + + **A capability this SDK cannot express is skipped first, and says so.** Its + scenarios would be skipped anyway -- nothing may declare it, so nothing + does -- but with the wrong reason. "The provider does not declare it" reads + as a decision the provider made, and no provider in this language had one to + make; a reader of the report has to be able to tell those apart, because only + the first says anything about the provider. Checked before the declaration + loop rather than inside it so that a scenario gated by both kinds reports the + permanent, language-wide reason rather than whichever tag came first off the + marker iterator. Empty in Python; see + :data:`~.capability.INEXPRESSIBLE_CAPABILITIES`. """ gated = [ capability @@ -235,17 +257,59 @@ def _tck_capability_gate(request: pytest.FixtureRequest) -> None: if not gated: return + inexpressible = inexpressible_skip_reason(gated) + if inexpressible is not None: + pytest.skip(inexpressible) + try: config: TckConfig = request.getfixturevalue("tck_config") except pytest.FixtureLookupError: return + undeclared = undeclared_skip_reason(gated, config) + if undeclared is not None: + pytest.skip(undeclared) + + +def inexpressible_skip_reason(gated: Iterable[Capability]) -> str | None: + """Why these scenarios cannot be run in this language at all, or ``None``. + + Says nothing about the provider, and says so, because the alternative + reading is the one a reader will reach for: a capability missing from a + report usually means the provider declined. Here nothing declined -- no + provider in this SDK could be asked -- and Appendix F makes telling those + two apart the implementation's job rather than the reader's. + + Deterministic when more than one applies: the tags are sorted, so the + message does not depend on the order markers come off a node. + """ + for capability in sorted(gated, key=lambda c: c.tag): + reason = capability.inexpressible_reason + if reason is not None: + return ( + f"{capability.tag} cannot be expressed by this SDK, so no provider " + f"in this language can be asked: {reason}. Nothing about the " + f"provider under test follows from this skip" + ) + return None + + +def undeclared_skip_reason( + gated: Iterable[Capability], config: TckConfig +) -> str | None: + """Why this provider is not being asked these scenarios, or ``None``. + + The other half of the pair, and the one that *is* about the provider: it + declined, and the declaration it did make is quoted so a reader can see what + was claimed instead. + """ for capability in gated: if not config.declares(capability): - pytest.skip( + return ( f"provider does not declare capability {capability.tag}. " f"Declared: {' '.join(config.sorted_capabilities) or '(none)'}" ) + return None @pytest.fixture(scope="session", autouse=True) diff --git a/tools/openfeature-tck/tests/test_declaration.py b/tools/openfeature-tck/tests/test_declaration.py index 857e05b23..d23d6fc87 100644 --- a/tools/openfeature-tck/tests/test_declaration.py +++ b/tools/openfeature-tck/tests/test_declaration.py @@ -29,6 +29,7 @@ from openfeature.contrib.tools.tck import ( DECLARABLE_CAPABILITIES, + INEXPRESSIBLE_CAPABILITIES, RESERVED_CAPABILITIES, BackendControl, Capability, @@ -39,6 +40,8 @@ canonical_root, plugin, ) +from openfeature.contrib.tools.tck import capability as capability_module +from openfeature.contrib.tools.tck import config as config_module from openfeature.contrib.tools.tck.capability import ( capability_for_marker, capability_for_tag, @@ -82,19 +85,32 @@ def _config(**overrides: typing.Any) -> TckConfig: # -- the vocabulary ---------------------------------------------------------- -def test_every_capability_is_either_declarable_or_reserved() -> None: - """Two sets, one enum, and no member in both or neither. +def test_every_capability_is_declarable_reserved_or_inexpressible() -> None: + """Three sets, one enum, and no member in two of them or in none. - ``DECLARABLE_CAPABILITIES`` is derived from ``RESERVED_CAPABILITIES`` rather - than listed beside it, so this is really a check that the derivation is the - one the documentation promises. + ``DECLARABLE_CAPABILITIES`` is derived from the other two rather than listed + beside them, so this is really a check that the derivation is the one the + documentation promises. It is worth pinning precisely because the third set + is empty here: a derivation that quietly dropped it would look right in + Python forever and be wrong the day an entry is added. + + Nothing may be both reserved and inexpressible. A reservation says no + scenario anywhere carries the tag; inexpressibility says the scenarios exist + and this SDK cannot put their question. The second presupposes what the first + denies. """ - assert frozenset(Capability) == DECLARABLE_CAPABILITIES | RESERVED_CAPABILITIES + inexpressible = frozenset(INEXPRESSIBLE_CAPABILITIES) + assert frozenset(Capability) == ( + DECLARABLE_CAPABILITIES | RESERVED_CAPABILITIES | inexpressible + ) assert not DECLARABLE_CAPABILITIES & RESERVED_CAPABILITIES + assert not DECLARABLE_CAPABILITIES & inexpressible + assert not RESERVED_CAPABILITIES & inexpressible assert RESERVED_CAPABILITIES, "the whole rule is vacuous if nothing is reserved" for capability in Capability: assert capability.reserved is (capability in RESERVED_CAPABILITIES) + assert capability.inexpressible is (capability in inexpressible) def test_a_reserved_capability_is_one_no_canonical_scenario_carries() -> None: @@ -419,6 +435,211 @@ def test_the_refusal_says_what_may_be_declared_instead() -> None: assert capability.tag in message +# -- what this SDK cannot express -------------------------------------------- +# +# INEXPRESSIBLE_CAPABILITIES is empty in Python, and that was measured: `int` is +# arbitrary-precision and `get_integer_details` and `get_float_details` are +# separate accessors reaching separate provider methods, so all four questions +# the two tagged groups ask can be put, and were. The machinery is here anyway, +# because the rule belongs to Appendix F rather than to this package and the next +# capability may hit it -- and a mechanism nothing exercises is indistinguishable +# from a mechanism that does not work. So these tests supply an entry rather than +# skipping for want of one, and the fabricated entry is Java's real case. + + +_AS_IN_JAVA = ( + "the integer accessor is a 32-bit Integer, so 2^53 - 1 cannot be asked for" +) + + +@pytest.fixture +def one_inexpressible(monkeypatch: pytest.MonkeyPatch) -> Capability: + """Pretend, for one test, that this SDK cannot express ``@large-integers``. + + Patched on the module rather than injected, because the production code + reads the mapping through the module global at call time and an injected + copy would test a seam nothing else uses. + """ + monkeypatch.setattr( + capability_module, + "INEXPRESSIBLE_CAPABILITIES", + types.MappingProxyType({Capability.LARGE_INTEGERS: _AS_IN_JAVA}), + ) + return Capability.LARGE_INTEGERS + + +def test_an_inexpressible_capability_is_one_whose_scenarios_exist() -> None: + """The property that tells it from a reservation, read off the assets. + + A capability nothing carries is reserved, whatever any SDK could express + about it -- so an entry here whose tag no canonical scenario carries is + misfiled, and the two would then differ only in their wording. Every entry + also has to say *which* property of the SDK puts the question out of reach, + because that is the half of the message an adopter could not have worked out. + + Vacuous while the mapping is empty, and kept for the pass where it is not. + """ + carried = canonical_tags() + for capability, reason in INEXPRESSIBLE_CAPABILITIES.items(): + assert capability.tag in carried, ( + f"{capability.tag} is recorded as inexpressible but no canonical " + f"scenario carries it, which makes it a reservation instead" + ) + assert reason.strip(), ( + f"{capability.tag} does not say what puts it out of reach" + ) + + +def test_a_capability_this_sdk_cannot_express_cannot_be_declared( + one_inexpressible: Capability, +) -> None: + """Refused by the implementation, rather than left for adopters to remember. + + Which is the whole change: the fact is about the language, so an adopter + should not have to know it, and three suites each remembering it separately + is three chances to put an unverifiable claim in a report. + """ + with pytest.raises(ValueError) as raised: + _config(capabilities={Capability.EVENTS, one_inexpressible}) + + message = str(raised.value) + assert f"{one_inexpressible.tag} cannot be declared in this language" in message + # The property of the SDK, not the rule. An adopter reaching this has done + # nothing wrong and needs to be told something they could not have known. + assert _AS_IN_JAVA in message + assert "nothing for you to fix" in message + + +def test_the_two_refusals_do_not_read_the_same(one_inexpressible: Capability) -> None: + """A reader has to be able to tell a reservation from an impossibility. + + Reserved: global, temporary, expires when the specification writes a + scenario. Inexpressible: one language's, permanent, and the scenarios + already exist and pass elsewhere. Both end in a refusal and nothing else + about them is the same, so neither message may be reachable from the other's + predicate. + """ + reserved = next(iter(sorted(RESERVED_CAPABILITIES, key=lambda c: c.tag))) + + with pytest.raises(ValueError) as raised: + _config(capabilities={reserved}) + reserved_message = str(raised.value) + + with pytest.raises(ValueError) as raised: + _config(capabilities={one_inexpressible}) + inexpressible_message = str(raised.value) + + assert "no scenario carries them" in reserved_message + assert "no scenario carries" not in inexpressible_message, ( + "its scenarios do exist -- that is what makes it not a reservation" + ) + assert "is not a reservation" in inexpressible_message + assert _AS_IN_JAVA not in reserved_message + + # And they are produced by separate predicates, so neither can start + # answering for the other. + assert config_module.reserved_problems([one_inexpressible]) == [] + assert config_module.inexpressible_problems([reserved]) == [] + + +def test_a_deviation_may_not_name_a_capability_this_sdk_cannot_express( + one_inexpressible: Capability, +) -> None: + """A deviation is about this provider; this gap belongs to the language. + + Refused for the opposite reason a reserved one is. There the scenarios do + not exist, so there is nothing to deviate from; here they exist and no + provider in this SDK can attempt them, so the entry would attribute to one + provider something none of them could have done. + """ + with pytest.raises(ValueError) as raised: + _config( + known_deviations=[ + KnownDeviation.untracked(summary="a gap", capability=one_inexpressible) + ] + ) + + message = str(raised.value) + assert f"names {one_inexpressible.tag}, which cannot be expressed" in message + assert _AS_IN_JAVA in message + assert "belongs to the language" in message + + +def test_the_skip_reason_says_no_provider_here_could_have_been_asked( + one_inexpressible: Capability, +) -> None: + """The second half, and the one a report's reader actually sees. + + Refusing the declaration is not enough on its own: the scenarios are skipped + either way, and a skip reading "provider does not declare capability + @large-integers" describes a decision no provider in this language had the + chance to make. + """ + reason = plugin.inexpressible_skip_reason([one_inexpressible]) + assert reason is not None + assert _AS_IN_JAVA in reason + assert "no provider in this language can be asked" in reason + assert "Nothing about the provider under test follows" in reason + + declined = plugin.undeclared_skip_reason( + [Capability.EVENTS], _config(capabilities=frozenset()) + ) + assert declined is not None + assert "provider does not declare capability @events" in declined + assert declined != reason + + +def test_the_gate_prefers_the_language_reason_over_the_declaration_one( + one_inexpressible: Capability, +) -> None: + """A scenario gated by both kinds reports the permanent one. + + ``@large-integers`` and ``@events`` on one scenario, neither declared: the + marker iterator decides which comes first, and the message must not. The + language-wide reason is the true one -- the provider's declaration could not + have made this scenario run. + """ + node = types.SimpleNamespace( + iter_markers=lambda: iter( + [ + types.SimpleNamespace(name=Capability.EVENTS.value), + types.SimpleNamespace(name=one_inexpressible.value), + ] + ) + ) + request = types.SimpleNamespace( + node=node, + getfixturevalue=lambda name: _config(capabilities=frozenset()), + ) + + with pytest.raises(pytest.skip.Exception) as raised: + plugin.capability_gate(typing.cast("pytest.FixtureRequest", request)) + + assert _AS_IN_JAVA in str(raised.value) + + +def test_the_gate_still_skips_for_a_declaration_nobody_made() -> None: + """The unchanged half, checked here because the branch above is new. + + Nothing is patched: ``@events`` is expressible and undeclared, which is the + ordinary case and the one that must keep its own wording. + """ + node = types.SimpleNamespace( + iter_markers=lambda: iter([types.SimpleNamespace(name=Capability.EVENTS.value)]) + ) + request = types.SimpleNamespace( + node=node, + getfixturevalue=lambda name: _config(capabilities={Capability.OBJECT}), + ) + + with pytest.raises(pytest.skip.Exception) as raised: + plugin.capability_gate(typing.cast("pytest.FixtureRequest", request)) + + message = str(raised.value) + assert "provider does not declare capability @events" in message + assert "Declared: @object" in message + + # -- acknowledging a gap ----------------------------------------------------- From 579796f052baad6f0f78e1d9cc2ab9e358136e2d Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 17:22:46 +0200 Subject: [PATCH 39/46] ci: let every matrix cell report, instead of cancelling on the first failure The build matrix is five Python versions times every package the change touched. The point of a matrix that wide is to say which combinations fail -- a package that breaks only on 3.10, a package that breaks everywhere, a package that is fine. `fail-fast` defaults to true, so the first cell to fail cancels all the others, and a run that could have reported thirty results reports one. The information lost is the part that costs the most to recover. A cancelled cell is not a passing cell and not a failing one; it is silence, and the only way to find out what it would have said is to push again and hope the cells finish in a different order. Worse, a failure in a package the pull request did not touch is indistinguishable from a failure in the package it did -- both present as "the build job failed", with most of the evidence cancelled. Observed rather than hypothesised, across two consecutive heads of one pull request: an unrelated package's cell failed, and every cell of two other packages was cancelled with it. At the earlier head four of five cells of one package completed; at the later head none did. That package's own results were never reported at all, for reasons that had nothing to do with it. The cost of the change is runner minutes spent on cells that were going to fail anyway. That is the trade this setting exists to make, and for a matrix whose whole purpose is telling combinations apart it is the right side of it. Signed-off-by: Simon Schrottner --- .github/workflows/build.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index d59c38f7d..f79c4405e 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -81,6 +81,12 @@ jobs: needs: changes runs-on: ubuntu-latest strategy: + # A matrix of every supported Python against every changed package exists + # to say *which* combinations fail. The default `fail-fast: true` discards + # exactly that: the first failing cell cancels every other one, so a run + # that could have reported thirty results reports one, and an unrelated + # package's failure is indistinguishable from your own. + fail-fast: false matrix: python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"] package: ${{ fromJSON(needs.changes.outputs.packages) }} From a7dd6c9a964d4aee27fb065a0992e9f5bc778e53 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 18:44:30 +0200 Subject: [PATCH 40/46] chore(tck): re-pin the conformance assets to spec 4cab0320 Two prose corrections to Appendix F and nothing else: 045950ca rewrites the numeric-coercion note that had been teaching withhold-plus-deviate, and 4cab0320 adds a sixth declaring rule saying the unit of a declaration decision is the scenario rather than the tag. `git diff --name-only 89b1519a 4cab0320` is one file, the appendix itself, and over `specification/assets/` it is empty -- the Gherkin, the canonical flag set and the control API document are byte-identical, and the synced copies under `src/` hash the same before and after. So no scenario count moves and none should: the self-tests stay at 221 passed, 42 skipped, 2 xfailed. The pin still has to move, because it is what records which questions a run asked, and a suite documenting rules from a revision it does not name is the drift this submodule exists to prevent. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/spec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/openfeature-tck/spec b/tools/openfeature-tck/spec index 89b1519a0..4cab03204 160000 --- a/tools/openfeature-tck/spec +++ b/tools/openfeature-tck/spec @@ -1 +1 @@ -Subproject commit 89b1519a08d81c46ba47fc2a54c44d40fdee845d +Subproject commit 4cab032043f9b2335a023310b1010b20d7a7d0d9 From fc720effbbf7613dda17cf82f3815dc65e0e7d46 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 18:44:45 +0200 Subject: [PATCH 41/46] docs(tck): stop illustrating a rule with the shape it forbids Three paragraphs here told an adopter that a provider which narrows 0.5 to 0 should withhold @numeric-coercion and record a deviation for it -- "withholding it may be a deliberate choice as readily as a known bug. Where it is a bug, say so" in the enum, "withholding the tag is the honest report for each" twice over about flagd's two resolvers. That is withhold-plus-deviate, which the known deviation rule two sections away tells an adopter to avoid, and it is not what this repository's own flagd adoption does: it declares the tag on both resolvers and puts the deviation on the one that narrows. The wording was inherited from Appendix F, which said the same thing until spec@045950ca corrected it. The distinction the paragraphs draw is worth keeping -- an undeclared capability can be a choice or a defect, and a report that cannot tell them apart is worth less -- so the distinction stays and the illustration is replaced by the one that matches: attempts the coercion and gets a direction wrong, declare and deviate; cannot attempt it at all, withhold. The SDK's in-memory provider is the second kind and says so. The paragraph about flagd also called both resolvers defective, which is false in the other direction: in-process refuses 0.5 correctly. One resolver of one implementation is wrong, which is the finding, and it survives only because the tag was declared. Also cites the sixth declaring rule (spec@4cab0320) where an adopter narrows the default capability set, rather than restating it -- that rule was written from the wording in this repository's flagd suites, and four independent statements of one rule is what this effort keeps having to undo. Its first consequence goes next to the field it concerns: a scenario failing for a fixture the backend does not serve is not a provider defect and does not belong in knownDeviations. And the two self-test withholdings that are withholdings for a defect now name the carve-out that licenses them (spec@045950ca) instead of arguing the case again, with the condition each meets. @disabled-flags meets it exactly: the sweep in test_in_process_control asserts that all four disabled flags resolve to their own default variant, so the behaviour is pinned by a test of its own and that test turns red the day the SDK honours DISABLED. @configuration-change meets it differently and the note says so -- the scenarios are not only skipped, test_controllable_conformance runs them against the subclass that supplies what the SDK lacks -- with the one thing that pin does not do stated plainly: nothing fails on its own when the SDK gains the method. An adoption has no such licence, and both notes say that too, because these files are the nearest worked example an adopter will copy from. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 52 +++++++++++++++---- .../contrib/tools/tck/capability.py | 30 +++++++---- .../openfeature/contrib/tools/tck/config.py | 8 +++ .../tests/test_in_memory_conformance.py | 32 ++++++++---- 4 files changed, 95 insertions(+), 27 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 7bb2ac0a8..19b7afada 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -358,6 +358,15 @@ Untagged scenarios are mandatory and always run. `capabilities` defaults to ever capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from the default, run the suite, and remove only what your provider genuinely cannot do. +**What counts as "cannot do" is [Appendix F][appendix-f]'s to say, and the rule is not about the tag +but about its scenarios**: declare a capability when at least one scenario gating it can actually be +put to the provider, and withhold it only when none can. Its two consequences are the ones that bite +in practice — a scenario that fails because the backend serves no fixture for it is not a provider +defect and must not be recorded as one, and a capability withheld for a backend gap is temporary in +a way one withheld by choice is not, so it needs a note saying why or it outlives its reason. The +flagd adoption in this repository decides `@numeric-coercion` and `@large-integers` by that rule and +gets opposite answers; its suite files cite it rather than restating it, and so should yours. + Leaving a capability out is the only way to withhold it, and one skip carrying its reason is the whole mechanism: the scenario's tags say what was asked, the declaration says whether it was claimed, and the skip says why it was not. @@ -384,10 +393,13 @@ declaring `@numeric-coercion` in both flagd suites and running it: flagd's in-pr refuses `0.5` as an integer and widens `10` to a float, and its RPC resolver widens `10` and silently narrows `0.5` to `0`, which is the one thing the lossy scenario forbids. Two different answers to the same three questions, from two resolvers of one provider — which is what a language -that *could not ask* them would make impossible. Both are defects in an implementation, withholding -the tag is the honest report for each, and neither has anything to do with Python. (The third -scenario fails on both for a third reason again: flagd-testbed does not seed `integral-float-flag` -at all.) +that *could not ask* them would make impossible. That split is a defect in one resolver of one +implementation, and **both** resolvers declare the tag: the one that narrows carries a +`KnownDeviation` and the one that does not carries none, which is the shape [Appendix F][appendix-f] +prefers and the opposite of what this paragraph used to prescribe. Neither has anything to do with +Python. (The third scenario fails on both for a third reason again: flagd-testbed seeds no +`integral-float-flag` at all — the backend's gap, not the provider's, and recorded as such rather +than as a reason to withhold.) **A reservation and an inexpressibility are not the same refusal**, and the messages and the skip reasons deliberately differ: @@ -447,8 +459,14 @@ It comes from flagd's [numeric coercion ADR](https://github.com/open-feature/flagd/blob/main/docs/architecture-decisions/numeric-coercion.md), which is scoped to flagd's own implementations, and the tag carries that name — it was `@strict-numeric-typing` — because two vocabularies for one observable property is worse than one -borrowed name. **A provider that behaves differently is not violating the specification**, so -withholding this capability may be a deliberate choice as readily as a defect. +borrowed name. **A provider that behaves differently is not violating the specification** — but that +does not leave a missing declaration free to interpret, and [Appendix F][appendix-f]'s note on this +tag says which is which. A provider that *attempts* the coercion and gets one direction wrong +declares the capability, lets the lossy scenario fail and records a `KnownDeviation` beside it, +because "it coerces, and one direction is wrong" is what a skip cannot say. Withholding is for a +provider that *cannot attempt* it: a language with a single numeric type, or one that hands every +variant back untouched and never coerces — which is what this SDK's `InMemoryProvider` does, and why +neither in-memory self-test declares the tag. Both halves are tested, and a provider declaring the tag must satisfy all three scenarios: `float-flag` (`0.5`) requested as an integer is a `TYPE_MISMATCH`; `integral-float-flag` (`10.0`) requested as an @@ -694,6 +712,14 @@ Only half the machinery is missing — `AbstractProvider` already supplies `emit_provider_configuration_changed` — which is why `ControllableInMemoryProvider` here is a small subclass rather than a reimplementation, and why it should port back to the SDK as a method. +`test_in_memory_conformance` therefore withholds `CONFIGURATION_CHANGE` for a defect, which again is +the [Appendix F][appendix-f] self-test carve-out rather than something an adoption may copy. It meets +the condition differently from finding 4: the scenarios are not only skipped, they are *run* — by +`test_controllable_conformance`, against the subclass that supplies what the SDK lacks, so the step +definitions and the change path stay covered and the gap is written down in `PlainMemoryControl`'s +`change_flag`, which raises rather than pretends. The one thing this pin does not do is go red on +its own when the SDK is fixed; nothing fails at that point, the subclass just becomes redundant. + ### 3. The in-memory provider does not coerce numbers `integral-float-flag` (`10.0`) requested as an integer returns the code default with `TYPE_MISMATCH`, @@ -722,9 +748,17 @@ Measured before the tag was gated: all four rows failed on the value in both in- `disabled-boolean-flag` resolving to `True` against a caller default of `false`. So neither suite declares `@disabled-flags` and the four scenarios are skipped with that reason. -Unlike finding 3 this is a field the SDK offers and does not honour, which is closer to a defect than -to a choice — but the capability is optional, so the honest report is still a withheld declaration -rather than a `KnownDeviation`. It is not filed against the SDK yet. +Unlike finding 3 this is a field the SDK offers and does not honour, so it is a defect rather than a +choice — and withholding a capability for a defect is the thing [Appendix F][appendix-f] tells an +*adoption* not to do. What licenses it here is the appendix's self-test carve-out: these two suites +are a fixture for the harness rather than a report about a third party, they run in the ordinary +build where a permanently failing scenario is a broken build rather than a finding, and the fix is an +SDK release away. **The carve-out's condition is that the defect is pinned by a test of its own, and +it is**: `test_every_packaged_flag_resolves_to_its_packaged_default_variant` sweeps the four +`disabled-*` flags with everything else and asserts that each resolves to its own default variant — +so the behaviour is asserted rather than merely skipped, and that test turns red the day the SDK +starts honouring `DISABLED`, which is when the capability becomes declarable here. It is not filed +against the SDK yet. ## Where the assets come from diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py index ec5629e56..0a0550400 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py @@ -192,10 +192,21 @@ class Capability(str, Enum): **A provider that behaves differently is not violating the specification.** So this is genuinely optional, rather than optional as a concession to a - defect: withholding it may be a deliberate choice as readily as a known bug. - Where it is a bug, say so -- a report's ``knownDeviations`` is for exactly - that, and flagd's instance is tracked as `open-feature/flagd#1996 - `_. + defect -- but *which* of those a missing declaration means is not a free + choice, and this paragraph used to say it was. Appendix F's numeric-coercion + note settles it: a provider that **attempts** the coercion and gets one + direction wrong declares the capability, lets the lossy scenario fail, and + records a :class:`~.config.KnownDeviation` beside the failure -- because "it + coerces, and one direction is wrong" is exactly what a skip cannot say. + flagd is that provider (`open-feature/flagd#1996 + `_), and the flagd + adoption declares the tag and deviates rather than withholding. + + Withholding is for a provider that **cannot attempt** the behaviour: a + language with a single numeric type, where the distinction does not exist to + get wrong, or a provider that hands every variant back untouched and never + coerces at all. The SDK's own ``InMemoryProvider`` is the second kind, and + the paragraph below is what that looks like. Both halves have scenarios, and a provider declaring the tag must satisfy all three. The lossy half asks for ``float-flag`` (``0.5``) as an integer @@ -491,11 +502,12 @@ def __str__(self) -> str: and belongs nowhere near this mapping. flagd's two Python resolvers answer the three ``@numeric-coercion`` scenarios differently from each other: in-process refuses ``0.5`` as an integer and widens ``10`` to a float, while RPC widens -``10`` and silently narrows ``0.5`` to ``0``. Each is a defect in an -implementation, recorded where that adoption records its defects, and -withholding the tag is the honest report for both. Listing it here would say the -question cannot be asked -- and two resolvers of one provider giving different -answers to it is the proof that it can. +``10`` and silently narrows ``0.5`` to ``0``. That split is a defect in one +resolver of one implementation, and **both** resolvers declare the tag: the one +that narrows carries the :class:`~.config.KnownDeviation` and the one that does +not carries none, which is the shape Appendix F prefers. Listing it here would +say the question cannot be asked -- and two resolvers of one provider giving +different answers to it is the proof that it can. Never overlaps :data:`RESERVED_CAPABILITIES`: a tag no scenario carries is reserved, whatever any SDK could express about it. diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py index 19e8f67de..3833d6445 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/config.py @@ -52,6 +52,14 @@ class KnownDeviation: the failure mode this field exists to prevent. If the provider attempts the behaviour and gets it wrong, shape 1 is the honest report. + **A scenario that fails because the backend serves no fixture for it is not a + provider defect and does not belong here.** That is the first consequence + Appendix F draws from its declaring rules, and an entry recording it would + accuse the provider of the backend's gap. Where such a failure sits under the + same tag as a real one -- which is the ordinary case, since the tag is + declared on the scenarios that *can* be asked -- say so in the summary of the + entry that covers the real one. + It changes nothing about how the suite runs. The scenario still fails, and the results payload still reports it as failed -- a report that softened a failure into a footnote would hide exactly what the acknowledgement exists to diff --git a/tools/openfeature-tck/tests/test_in_memory_conformance.py b/tools/openfeature-tck/tests/test_in_memory_conformance.py index aef4a69bc..03bba6150 100644 --- a/tools/openfeature-tck/tests/test_in_memory_conformance.py +++ b/tools/openfeature-tck/tests/test_in_memory_conformance.py @@ -83,7 +83,11 @@ def tck_config() -> TckConfig: * ``CONFIGURATION_CHANGE`` -- omitted because the SDK's in-memory provider cannot update its flag set. That is a finding, not a configuration choice; - see ``PlainMemoryControl``. + see ``PlainMemoryControl``. Withheld for a defect under the same Appendix F + self-test carve-out as ``DISABLED_FLAGS`` below, and pinned differently: + ``test_controllable_conformance`` *runs* these scenarios against + ``ControllableInMemoryProvider``, which supplies what the SDK lacks, so the + skip here is not the only record of them. * ``STALE`` and ``UNAVAILABLE_INIT`` -- omitted because there is no connection to lose. ``PlainMemoryControl`` does not implement ``ConnectionControl`` for the same reason, and the two omissions keep each @@ -116,11 +120,20 @@ def tck_config() -> TckConfig: flags are served at their own default variant with reason ``STATIC``, where the scenarios expect the caller's default. Measured before it was gated: the four rows failed on the value, ``disabled-boolean-flag`` - resolving to ``True`` against a caller default of ``false``. The - capability is optional, so the honest report is a withheld declaration - rather than a ``KnownDeviation`` -- but unlike ``NUMERIC_COERCION`` this - one is a field the SDK offers and does not honour, which is finding 4 in - the README. + resolving to ``True`` against a caller default of ``false``. + + Unlike ``NUMERIC_COERCION`` this is a field the SDK offers and does not + honour, so it is a defect (finding 4 in the README) -- and **withholding a + capability for a defect is what Appendix F's self-test carve-out + licenses, not something an adoption may copy**. This suite is a fixture + for the harness rather than a report about a third party, and it runs in + the ordinary build where a permanently failing scenario is a broken build + rather than a finding. The carve-out's condition is that the defect be + pinned by a test of its own, and it is: + ``test_every_packaged_flag_resolves_to_its_packaged_default_variant`` + sweeps the four flags with the rest and asserts each resolves to its own + default variant, so the behaviour is asserted rather than only skipped and + the sweep turns red the day the SDK honours ``DISABLED``. * ``NUMERIC_COERCION`` -- omitted because the SDK's in-memory provider does not coerce. It hands each variant back untouched, and the client's type check is ``isinstance``-based, so ``integral-float-flag`` (``10.0``) @@ -128,9 +141,10 @@ def tck_config() -> TckConfig: ``integer-flag`` (``10``) requested as a float is one rather than ``10.0``. The lossy scenario passes for the wrong reason -- every float is rejected -- which is exactly what the two lossless scenarios exist to - catch, and declaring the tag would have them catch it here. The - capability is optional, so this is a choice the provider is entitled to - rather than a deviation. + catch, and declaring the tag would have them catch it here. This is the + withholding Appendix F still calls right after its correction: a provider + that does not attempt the coercion at all, rather than one that attempts + it and gets a direction wrong. It needs no carve-out and no deviation. ``LARGE_INTEGERS`` is declared: a Python ``int`` is unbounded and nothing in this provider routes a value through a float. ``VARIANTS`` is declared From 20248774b6c381dce555f32047ed0bdd42c19055 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 19:13:37 +0200 Subject: [PATCH 42/46] docs(tck): follow rule six's correction, and illustrate the choice/defect split Re-pins to spec@aa2ad24f, which narrows the rule this repository's flagd suites supplied. As first written it said "declare when at least one scenario gating it can be put to the provider" with no condition, which read literally forces a declaration wherever the scenarios are reachable -- including where the specification permits declining outright, and @numeric-coercion is exactly that case. The rule now opens with the condition it always had implicitly: it applies once a provider is attempting the capability, and whether the provider owes an answer at all is the known-deviation rule's question, asked first. Assets are byte-identical across all three revisions, so nothing moves. The README's citation carries the two questions in that order rather than the second one alone, because the second alone is the over-reach. `DISABLED_FLAGS` was a fourth site teaching the shape the appendix corrected, and the grep for the numeric-coercion wording did not find it: "withholding it still needs no KnownDeviation ... that holds whether the gap is architectural or a defect; where it is a defect, the adoption's note is where to say so". The two cases are not alike. A backend that gives the provider no signal is a withholding with nothing to record; a provider one unconditional index from passing -- which is the Python OFREP provider, named two paragraphs above -- is the declare-and- deviate case, and sending both to "the adoption's note" is how a defect ends up looking deliberate. `CONFIGURATION_CHANGE` gains the illustration the numeric-coercion note used to carry badly: **one capability withheld twice in this repository for two different reasons.** The OFREP adoption withholds it by choice -- no stream, no poll, nothing watching, and no defect in building a provider that way -- and the in-memory self-test withholds it because the SDK's provider cannot update its flag set at all, which Appendix A requires of it (python-sdk#620). Identical in the results, distinguishable only from what the adoption wrote down, and both true here rather than hypothetical. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 30 +++++++++---- tools/openfeature-tck/spec | 2 +- .../contrib/tools/tck/capability.py | 45 ++++++++++++++++--- 3 files changed, 62 insertions(+), 15 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 19b7afada..638509730 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -358,19 +358,33 @@ Untagged scenarios are mandatory and always run. `capabilities` defaults to ever capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from the default, run the suite, and remove only what your provider genuinely cannot do. -**What counts as "cannot do" is [Appendix F][appendix-f]'s to say, and the rule is not about the tag -but about its scenarios**: declare a capability when at least one scenario gating it can actually be -put to the provider, and withhold it only when none can. Its two consequences are the ones that bite -in practice — a scenario that fails because the backend serves no fixture for it is not a provider -defect and must not be recorded as one, and a capability withheld for a backend gap is temporary in -a way one withheld by choice is not, so it needs a note saying why or it outlives its reason. The -flagd adoption in this repository decides `@numeric-coercion` and `@large-integers` by that rule and -gets opposite answers; its suite files cite it rather than restating it, and so should yours. +**What counts as "cannot do" is [Appendix F][appendix-f]'s to say, and there are two questions in it, +asked in order.** First: does your provider owe an answer at all? Where the specification permits +declining — `@numeric-coercion` is defined by no requirement, so a provider that simply does not +coerce is entitled to withhold it — withholding is the honest report however reachable the scenarios +are. Only once a provider *is* attempting the capability does the second question arise, and it is +not about the tag but about its scenarios: declare when at least one scenario gating it can actually +be put to the provider, and withhold only when none can. + +That second rule's two consequences are the ones that bite in practice — a scenario that fails +because the backend serves no fixture for it is not a provider defect and must not be recorded as +one, and a capability withheld for a backend gap is temporary in a way one withheld by choice is +not, so it needs a note saying why or it outlives its reason. The flagd adoption in this repository +decides `@numeric-coercion` and `@large-integers` by it and gets opposite answers; its suite files +cite it rather than restating it, and so should yours. Leaving a capability out is the only way to withhold it, and one skip carrying its reason is the whole mechanism: the scenario's tags say what was asked, the declaration says whether it was claimed, and the skip says why it was not. +**The same skip can mean two different things, so say which in your own note.** Both are in this +repository, on one capability: the OFREP adoption withholds `@configuration-change` because nothing +watches the backend — every evaluation is an independent request, and a provider built that way is +not defective — while the in-memory self-test withholds it because the SDK's provider cannot update +its flag set at all, which [Appendix A][appendix-a] requires of it +([python-sdk#620](https://github.com/open-feature/python-sdk/issues/620)). A choice and a defect, +identical in the results, distinguishable only from what the adoption wrote down. + ### A capability this SDK cannot express Some capabilities cannot hold in a language *at all* — `@numeric-coercion` where the language has a diff --git a/tools/openfeature-tck/spec b/tools/openfeature-tck/spec index 4cab03204..aa2ad24f5 160000 --- a/tools/openfeature-tck/spec +++ b/tools/openfeature-tck/spec @@ -1 +1 @@ -Subproject commit 4cab032043f9b2335a023310b1010b20d7a7d0d9 +Subproject commit aa2ad24f5a14ae2b5756df0b6d23f493f39507e6 diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py index 0a0550400..0830465b6 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py @@ -62,7 +62,32 @@ class Capability(str, Enum): """Provider enters ``STALE`` and emits ``PROVIDER_STALE`` when it loses its backend.""" CONFIGURATION_CHANGE = "configuration-change" - """Provider detects configuration changes and emits ``PROVIDER_CONFIGURATION_CHANGED``.""" + """Provider detects configuration changes and emits ``PROVIDER_CONFIGURATION_CHANGED``. + + **The worked example of one skip meaning two different things**, which is + why it is spelled out on this tag rather than left abstract. Both of this + repository's withholdings of it are real and neither is the other: + + * **A choice.** The OFREP adoption withholds it because every evaluation is + an independent HTTP request: there is no stream, no poll and no background + thread, so nothing is watching the backend and there is nothing to notice. + A provider built that way is not defective, and a + :class:`~.config.KnownDeviation` there would assert a defect that does not + exist. The next evaluation does return the new value -- what is missing is + the *signal*, which is what the scenario asserts. + * **A defect.** The in-memory self-test withholds it because the SDK's + ``InMemoryProvider`` copies its flag mapping in the constructor and exposes + no way to change it, and `Appendix A + `_ + **requires** an SDK's in-memory provider to support updating the flag set + and emitting this event. Tracked as `open-feature/python-sdk#620 + `_. + + A report shows the same absence in both cases, which is the whole reason the + declaration is not the last word: the adoption's note says which, and for the + second kind there is a ``ControllableInMemoryProvider`` here supplying what + the SDK lacks, so the scenarios still run somewhere. + """ OBJECT = "object" """Provider supports structured (object) flag values.""" @@ -127,11 +152,19 @@ class Capability(str, Enum): read as: not necessarily an impossibility, so a reader has to look at the adoption's own note for which it was. - Withholding it still needs no :class:`~.config.KnownDeviation`, for the - reason every gated capability does -- a deviation records a gap in behaviour - the provider is *required* to have, and this one is optional. That holds - whether the gap is architectural or a defect; where it is a defect, the - adoption's note is where to say so. + Withholding it **because the backend gives the provider nothing to act on** + needs no :class:`~.config.KnownDeviation`: a deviation records a gap in + behaviour the provider is *required* to have, and this one is optional, so an + entry would assert a defect that does not exist. + + A defect is the other case and does not get the same treatment, which this + docstring used to blur by sending both to "the adoption's own note". Where + the provider does attempt the resolution and gets it wrong -- the Python + OFREP provider above is exactly that, one unconditional index away from + passing -- Appendix F prefers the tag declared, the scenarios left to fail + and the deviation recorded beside them, because withdrawing the tag turns a + specific defect into a skip that reads as a design decision. Withholding is + for the provider that cannot attempt the behaviour at all. Nothing in the specification says what a provider owes a disabled flag. `Requirement 1.4.7 From f2c9f729f3dbb38c0eecc19e419b2c70ba14f221 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 20:58:56 +0200 Subject: [PATCH 43/46] docs(tck): rewrite the README as a guide to this binding This file is the package page on PyPI, and it was 59,708 bytes -- four times Appendix F, which is the normative document it implements. Someone arriving to adopt the suite had to read past four findings, the capability rationales, the control-API invariants and the decision history to reach a forty-line example. A base README documents how to use this library. Anything equally true of another language's binding belongs in Appendix F with a pointer, so it goes: the per-capability reasoning, the rules for declaring, the two shapes of a known deviation, the control-API invariants, the reserved/inexpressible comparison, and the findings, which are recorded in their filed issues and in the findings table on open-feature/spec#417. What stays is what is Python's. The adoption surface leads, because it is the shortest of the four -- two fixtures, one scenarios() call, and a step vocabulary arriving through a pytest11 entry point, so there is no conftest.py at all. Then the full option surface as two reference tables, the capability vocabulary, the dedicated poe task, the extension point, and the gotchas that are genuinely this language's: bool subclassing int, the three steps that reach the provider directly, the SDK's InMemoryProvider, and the submodule sync. Two corrections on the way past. The poe snippet was stale -- it showed `test` and `test-cov` as direct pytest commands and omitted `test-tck-collect`, so an adopter copying it got the exclusion without the compile check, which is the opposite of what the prose fifteen lines below it described. It is now the real one, from the two adopting pyprojects. And finding 4 said the DISABLED defect was unfiled; it is python-sdk#627. No behaviour change: 221 passed, 42 skipped, 2 xfailed, unmoved. 59,708 bytes to 17,464. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 987 +++++++------------------------- 1 file changed, 209 insertions(+), 778 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 638509730..a54c36c62 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -1,33 +1,24 @@ # OpenFeature TCK (Python) -A conformance suite any OpenFeature Python provider can adopt to verify that it implements the +A conformance suite any OpenFeature Python provider can adopt to check that it implements the provider contract of the specification. -Named `tck` rather than `provider-tck` because the name should say what the package *is*, not what -its current contents test: the entry point is options-shaped, so a suite for something other than a -provider can join it later instead of a second package duplicating the harness. +It is the Python implementation of [Appendix F][appendix-f], which defines the Gherkin scenarios, the +canonical flag set and the control API every language's TCK runs, and carries the reasoning behind +all of it. **This README documents the Python binding and nothing else**; where a question is not +Python's, it links there. -OpenFeature's central promise is that swapping providers does not change application behaviour. -Nothing verifies that today, and every provider tests differently — so "implements the provider -contract" is an unverified claim, and a behavioural difference between two providers is discovered -by the application that trips over it. +Tracking issue: [open-feature/spec#417][tracking]. **Status: proof of concept**, so expect breaking +changes. The one known gap that is this package's rather than the suite's: no multi-provider suite, +because the Python SDK has no multi-provider. -This package is the Python implementation of [Appendix F][appendix-f]. It runs the same Gherkin -scenarios, against the same canonical flag set, driven through the same backend control API, as -every other language's TCK. That shared basis is the point: "conformant" only means something if the -question is identical everywhere. +## Quick start -Tracking issue: [open-feature/spec#417][tracking]. +Two fixtures and one call. -## Status - -**Proof of concept.** The scenario set is a representative subset covering each architectural -mechanism once, not exhaustive coverage. Breaking changes should be expected. - -## Adopting it - -Two fixtures and one call. It uses **pytest-bdd**, the same runner the flagd provider and the flagd -testkit already use, so an adopting package gains no new test framework. +```bash +pip install 'openfeature-tck[compose]' +``` ```python import pytest @@ -66,150 +57,95 @@ def tck_config(tck_backend: RunningBackend): scenarios(*feature_paths()) ``` -**The suite owns the container stack.** You name a Compose file, say which ports the provider -connects to, and build a provider from the endpoint you are handed. Starting the stack, discovering -the dynamically mapped host ports, building the HTTP control against the control API, waiting until -it accepts commands and tearing down afterwards are all the suite's - see -[The container stack](#the-container-stack). A provider with **no** backend supplies a -`BackendControl` of its own instead and needs no Compose file and no container tooling - see -[Providers with no backend](#providers-with-no-backend). - -There is **no `conftest.py` to write and nothing to import for the steps**. The step definitions -arrive through this package's pytest plugin, registered via a `pytest11` entry point, so installing -the package is all it takes. +That is the whole adoption. **There is no `conftest.py` to write and nothing to import for the +steps** — they arrive through this package's pytest plugin, registered via a `pytest11` entry point — +and the assets ship in the distribution, so **adopting needs no git submodule**. The runner is +**pytest-bdd**, which the flagd provider and testkit already use, so an adopting package gains no new +test framework, and one test is generated per scenario and per Scenario Outline row. -The TCK owns the whole lifecycle: registering the provider under a suite-scoped domain, awaiting -events, resetting the backend between scenarios, releasing it at the end. **If you find yourself -writing test infrastructure, that is a defect here rather than something for you to work around.** +The suite owns the container stack and the provider lifecycle: starting Compose, discovering the +mapped host ports, building the HTTP control, registering the provider, awaiting events, resetting +the backend between scenarios, tearing down. **If you find yourself writing test infrastructure, +that is a defect here rather than something for you to work around.** -pytest-bdd generates one test per scenario — and one per row of a Scenario Outline — so failures -name a scenario and `-k` selects one as usual. The feature files and canonical flag set are packaged -inside the distribution, so **adopting this package needs no git submodule** — see -[Where the assets come from](#where-the-assets-come-from). +### A provider with no backend -### Timings +An in-memory, environment-variable or file-based provider supplies a `BackendControl` of its own, +where flag operations are direct manipulations of the provider's state. No Compose file and no +container tooling: `testcontainers` is the optional `compose` extra, imported lazily. -`TckConfig.event_timeout` is the knob that matters. Providers observe backend changes on wildly -different timescales — a streaming provider sees a configuration change in milliseconds, one that -polls every 30 seconds may need most of a poll interval. Set it to comfortably exceed your -worst-case detection latency, or the suite reports timeouts that are really just impatience. - -### Running it in CI - -**Keep the adoption suite out of the default build, give it a task of its own, and write down that -you did.** Why, and the two ways it goes wrong, are in Appendix F's ["Running the suite in -CI"][appendix-f]. It is not restated here: this section used to carry the reasoning in its own -words, in four languages, and that is where the same decisions came to have three different answers. - -The part that is Python's, and so belongs here — two `poe` tasks and one that CI never calls: - -```toml -[tool.poe.tasks] -test = "pytest tests --ignore=tests/tck" -test-cov = "coverage run -m pytest tests --ignore=tests/tck" -test-tck = "pytest tests/tck" +```python +@pytest.fixture(scope="session") +def tck_config(): + control = InProcessControl() + return TckConfig( + name="my-provider", + control=control, + new_provider=control.new_provider, + capabilities={Capability.EVENTS, Capability.OBJECT}, + ) ``` -`--ignore` on both of the tasks `build.yml` reaches — it runs `poe cov`, which is `test-cov` plus a -coverage report — and a comment above them saying why, so the exclusion cannot read as an oversight. -Both adoptions in this repository are exactly that, and each records its current tally in its own -README so a reviewer running `poe test-tck` can tell a new failure from a known one. - -Two Python-specific notes on top of the appendix: - -- **Docker is not what decides it here.** `tests/e2e` needs Docker too, has needed it for years, and - still runs in the default build on `ubuntu-latest`. The exclusion rests entirely on the second - reason, that the suite's honest output is red. -- **`--ignore` does not import the suite, so nothing checks that it still would.** `mypy` in these - packages is configured over `src` alone. So the default build also collects the excluded suite - without running it — `pytest tests/tck --collect-only` imports every test module, resolves the - feature files and starts no container — which is the appendix's "keep it compiling" in the form - Python has available. - -## Adding your own scenarios - -A provider is rarely only a provider. flagd has `fractional` targeting, another vendor has a -proprietary rollout rule, and the behaviour of those is as worth pinning as the contract they sit on -top of. Verifying them used to mean a second harness: a second backend lifecycle, a second set of -fixtures, a second thing to keep working. +It is a narrow allowance — **a provider with an external backend must drive it over the HTTP control +API**, for which `HttpControl` is the client, built on `urllib.request` alone so the TCK gains no +HTTP dependency; `control_api_spec()` returns the document a backend under test implements. Every +control states which path it took: `control_api` is a **required** member of `BackendControl`, typed +`Literal["http", "in-process"]`, with no default and no inference from the concrete type. A +backend-less one does not implement `ConnectionControl`, so `STALE` and `UNAVAILABLE_INIT` go +undeclared and their scenarios skip. -Put them in the same run instead. Create a directory named `extensions` beside the module that -calls `scenarios()`, and write step definitions for whatever is new in a `conftest.py` beside it: +## The options -``` -tests/ -├── conftest.py # your step definitions -├── test_conformance.py # the fixture and the one call, unchanged -└── extensions/ - └── fractional.feature -``` +### `TckConfig` -```python -# conftest.py -from pytest_bdd import then +| field | required | default | meaning | +| --- | --- | --- | --- | +| `name` | yes | — | the provider's name, as it appears in a report | +| `control` | yes | — | the `BackendControl` the scenarios drive the backend through | +| `new_provider` | yes | — | builds the provider under test, configured but uninitialised. Called **once per scenario**, because the mapped ports do not exist until the stack is up | +| `new_unavailable_provider` | no | `None` | builds a provider pointed at a closed port, for the initialisation-failure scenarios. Needed only if `capabilities` includes `UNAVAILABLE_INIT`; give it a short connection deadline, since the scenario allows a bounded time for the error | +| `capabilities` | no | `DECLARABLE_CAPABILITIES` | which optional parts of the contract this provider supports | +| `known_deviations` | no | `()` | gaps the provider is known to have | +| `event_timeout` | no | `12.0` | seconds to wait for a provider event | +| `ready_timeout` | no | `30.0` | seconds to wait for `READY`, and the longest a direct `shutdown`/`initialize` is given before the wait is recorded as a failure | -from openfeature.contrib.tools.tck import TckState +`event_timeout` is the knob that matters: set it comfortably above your provider's worst-case +detection latency — a poller may need most of a poll interval — or the suite reports timeouts that +are really just impatience. +### `ComposeBackend` -@then("the fractional rule splits the population") -def fractional_splits(tck_state: TckState) -> None: ... -``` +| field | required | default | meaning | +| --- | --- | --- | --- | +| `compose_file` | yes | — | path to the Compose file, resolved relative to the package directory | +| `backend_ports` | yes | — | container-internal ports the **provider** connects to. The control port is exposed automatically and must not be listed here | +| `backend_service` | no | `"backend"` | the Compose service hosting both the control API and the backend | +| `control_port` | no | `8080` | container-internal port of the control API | +| `additional_ports` | no | `{}` | extra service to ports, for a stack with more than one service. Resolved through the endpoint by service name | +| `backend_configuration` | no | `"default"` | the configuration name passed to `POST /start` | +| `startup_timeout` | no | `60.0` | seconds to wait for the stack and its control API to become reachable | -That is the whole of it — **no registration, no option and no new argument**. pytest collects -`conftest.py` on its own, pytest-bdd resolves steps through the fixture system, and the canonical -step vocabulary is in scope in your feature file beside your own steps. `tck_state` is the same -per-scenario state the canonical steps use, so your scenario runs against the provider the suite -registered, in the same backend lifecycle, with the same reset between scenarios. +These names and defaults are fixed across all four languages' TCKs, so a provider shipped in two of +them writes one Compose file and two declarations against it. -The one thing pytest cannot find by itself is the feature files, because the canonical ones are -inside the installed distribution rather than in your repository. `feature_paths()` returns both: +`tck_backend` is a session-scoped fixture the plugin builds from your `compose_backend` fixture, +yielding `.control` — the `HttpControl`, already awaited ready, and one per stack, since it remembers +whether a disconnect left the backend down — and `.endpoint`, with `host`, `port(internal)` and +`port(internal, service=...)`. Your Compose file must **not pin host ports**; declaring +`backend_ports` is what lets the harness say "the Compose file does not publish 8013" at startup +rather than three scenarios later. For a different fixture name or scope: ```python -scenarios(*feature_paths()) +@pytest.fixture(scope="session") +def tck_backend(): + yield from run_compose_backend(ComposeBackend(...)) ``` -That line does not change when you add an extension. An adopter with no `extensions` directory gets -the canonical set alone, so adding one is a matter of creating a directory rather than of -configuring anything. - -There used to be a second call, `features_path()`, which returned the canonical set on its own. It -is **gone**. The two differed by one character at the call site and the shorter one silently dropped -the extensions directory, so reaching for it produced a green run over fewer scenarios than the -adopter believed had run — which is the worst failure mode available to a conformance suite, because -nothing is there to notice. `canonical_root()` is the supported way to reach the packaged directory -for anything that is not "the scenarios to run". - -### Your scenarios cannot stand in for ours - -Every feature file carries a uri, and it is how a canonical scenario is told from an adopter's: -canonical files are the ones under the `gherkin/` prefix and yours are under `extensions/` — the -prefix Go and JavaScript mount theirs under too, so a consumer holding conformance reports from -several languages applies one rule. Neither prefix is this package's to choose: Appendix F -identifies a canonical feature by its path *relative to the specification's asset directory*, which -is what makes it `gherkin/`. The prefix is derived from where a file *is*, not from what the runner -called it, and `extensions.py` reports two cases that derivation cannot rule out: - -- **A feature file of yours under the reserved `gherkin/` prefix.** Handing `scenarios()` a - directory of your own named `gherkin` is the one route left to a canonical-looking uri. -- **Two feature files that would share one uri.** A record of what ran holds one copy of a feature - file per uri, so the second file's scenarios would be attributed to the first file's. - -This is not hypothetical. Java's suite found that a same-named feature file in a second classpath -root *replaced* the canonical one, and the run went green having asked the adopter's questions -instead of the specification's — the worst outcome available to a conformance suite. The Python -route to the same place is narrower and just as quiet: pytest-bdd names a feature file by its parent -directory joined to its own name, so `extensions/gherkin/errors.feature` arrives under the uri -the canonical `errors.feature` already occupies. - -## Capabilities - -Not every provider implements every optional part of the contract. Each scenario exercising an -optional part carries a Gherkin tag, pytest-bdd turns that tag into a pytest marker, and a provider -declares what it supports. - -**A scenario whose capability was not declared is reported as skipped, with the reason — never as -passed.** A conformance suite that quietly goes green on scenarios it did not run is worse than no -suite at all, so `pytest.skip` carries the reason into the report: +## Declaring capabilities + +Each scenario exercising an optional part of the contract carries a Gherkin tag, pytest-bdd turns it +into a marker, and a provider declares what it supports. **An undeclared capability's scenarios are +reported as skipped with the reason — never as passed:** ``` SKIPPED provider does not declare capability @stale. @@ -218,305 +154,38 @@ SKIPPED provider does not declare capability @stale. | Capability | Tag | Meaning | | --- | --- | --- | -| `Capability.LIFECYCLE` | `@lifecycle` | reaches its backend during initialisation, observably and promptly | +| `Capability.LIFECYCLE` | `@lifecycle` | reaches its backend during initialisation, observably | | `Capability.EVENTS` | `@events` | emits lifecycle events at all | | `Capability.STALE` | `@stale` | enters `STALE` and emits `PROVIDER_STALE` on backend loss | -| `Capability.CONFIGURATION_CHANGE` | `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | +| `Capability.CONFIGURATION_CHANGE` | `@configuration-change` | emits `PROVIDER_CONFIGURATION_CHANGED` on a configuration change | | `Capability.OBJECT` | `@object` | supports structured flag values | -| `Capability.VARIANTS` | `@variants` | names the variant it resolved, which [Requirement 2.2.4](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) makes a `SHOULD` and `types.md` types as optional | -| `Capability.DISABLED_FLAGS` | `@disabled-flags` | resolves a flag disabled in the management system to the code default | -| `Capability.UNAVAILABLE_INIT` | `@unavailable` | reports an error state instead of hanging against a dead backend | -| `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces between integer and float only when lossless, else `TYPE_MISMATCH` | -| `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly; undeclarable where the SDK's integer accessor is 32-bit | -| `Capability.REINITIALIZATION` | `@reinitialization` | can be initialised again after `shutdown`, which [Requirement 2.5.2](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) permits rather than requires | -| `Capability.TARGETING` | `@targeting` | resolves a flag differently for a matching evaluation context | -| `Capability.STANDARD_REASONS` | `@standard-reasons` | reports the standard resolution reasons, with the meanings [Appendix F][appendix-f] gives them | +| `Capability.VARIANTS` | `@variants` | names the variant it resolved | +| `Capability.DISABLED_FLAGS` | `@disabled-flags` | resolves a disabled flag to the code default | +| `Capability.UNAVAILABLE_INIT` | `@unavailable` | errors rather than hangs against a dead backend | +| `Capability.NUMERIC_COERCION` | `@numeric-coercion` | coerces int/float only when lossless, else `TYPE_MISMATCH` | +| `Capability.LARGE_INTEGERS` | `@large-integers` | resolves integers up to 2^53 − 1 exactly | +| `Capability.REINITIALIZATION` | `@reinitialization` | can be initialised again after `shutdown` | +| `Capability.TARGETING` | `@targeting` | resolves differently for a matching evaluation context | +| `Capability.STANDARD_REASONS` | `@standard-reasons` | uses the standard resolution reasons with their standard meanings | | `Capability.CACHING` | `@caching` | reserved; **not declarable** — no scenarios yet | -`@lifecycle` and `@events` are deliberately separate, and the split matters in both directions. An -SDK dispatches `PROVIDER_READY` around `initialize` for *any* provider, so a provider declaring only -`@events` passes the readiness scenario without demonstrating anything — a `NoOpProvider` passes it -identically. Meanwhile a stateless provider has a real initialisation to verify but no event stream -of its own to declare `@events` for, and gating on `@events` shut it out of a scenario it should be -held to. - -`@reinitialization` is separate from `@lifecycle` for a subtler reason. -[Requirement 2.5.2](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) -says a provider **SHOULD** revert to its uninitialized state after `shutdown`, and its supporting -text adds that *"some providers **may** allow reinitialization from this state"*. Reuse is therefore -permitted, not required: a provider that releases its client on shutdown and declines to be started -again is exercising a choice the specification offers it, so withholding this capability needs no -`KnownDeviation` entry. - -The scenario was untagged until spec revision `fc99d5ac`, on the reading that reverting to the -uninitialized state is observable as exactly one thing — being initialisable again. That inference -does not hold, and asserting it unconditionally reported a permitted choice as a conformance failure. -A false failure is the mirror image of a vacuous pass, and this suite cares about both. Reverting the -state is not separately observable either — a provider that reverts but refuses reuse presents -identically to one that did neither — so the gated reuse scenario is the only assertion the -requirement admits. It is worth keeping for the providers that do offer reuse, because releasing the -client on shutdown while leaving an initialised flag set behind is easy to write and leaves the -provider evaluating against a closed connection rather than failing outright. - -One practical note, because it is easy to get wrong: `@reinitialization` **narrows** `@lifecycle` -rather than standing beside it. The scenario lives in `lifecycle.feature`, which carries `@lifecycle` -at the feature level, so the scenario inherits it and carries both tags — and the gate skips a -scenario when *any* capability gating it is undeclared. Reuse is therefore exercised only by an -adoption declaring `Capability.LIFECYCLE` **and** `Capability.REINITIALIZATION`; declaring the latter -alone leaves the scenario skipped on `@lifecycle` and the declaration unverified. So a provider that -withholds `LIFECYCLE` has never run this scenario, and has no evidence either way on which to declare -reuse. - -`@variants` was the one found the hard way, and it is the reason the rule above is worth stating -twice. Every evaluation scenario used to assert a variant, which reads as obviously correct until a -backend with no variant concept for a plain flag is put under test: its evaluation response carries -no such key, the provider never receives one, and no seeding can produce one. Ten scenarios failed a -conformant provider for something its author could not fix, and nothing could be recorded as a -`KnownDeviation` because there was no capability to hang one on. -[Requirement 2.2.4](https://github.com/open-feature/spec/blob/main/specification/sections/02-providers.md) -is a **SHOULD** and `types.md` types the field `variant (string, optional)`, so the suite was -asserting a `MUST` neither of them states. Since spec revision `26362f85` the variant assertions -live in one gated Scenario Outline of eight rows; the value assertions stay untagged, because 2.2.3 -makes the value a `MUST`. - -`@targeting` was **reserved and undeclarable** until the same revision, on the reading that targeting -is backend evaluation logic and out of scope. The scope argument still holds — its three scenarios do -not test how a backend evaluates a rule — but the conclusion did not: they exist to show the context -reached the backend at all, which is a property of the provider and of nothing else. -`targeting-key-flag` is the one flag in the canonical set with a rule, specified by behaviour rather -than syntax (resolve `hit` when the targeting key is exactly `5c3d8535-f81a-4478-a6d3-afaa4d51199e`), -and a matching context resolving to a different value is what catches a provider that drops the -context — no echo endpoint on the control API required. The three scenarios are the matching context, -the non-matching one and no context at all; the second and third are not padding, since a provider -that always returned the targeted value would pass the first and one that refuses to evaluate a rule -with no targeting key is caught by the third. - -`@disabled-flags` is gated because it needs two things and only one of them comes for free. The -caller's default value is held by the provider, which always has it. What the provider also needs is -a **signal** that the flag was disabled, told apart from an ordinary resolution and from a missing -flag — and that belongs to the backend and its protocol. One with no disabled state, or one that -answers `FLAG_NOT_FOUND` for a disabled flag, gives the provider nothing to act on. - -[Appendix F][appendix-f] draws the line elsewhere — a provider whose backend decides, *"such as one -speaking OFREP, cannot: the server never sees the caller's default, so it has no way to return it"* — -and what this suite measured does not bear that out. flagd's RPC resolver is a remote evaluator by -exactly that description and satisfies the capability: the server answers reason `DISABLED` with no -variant and no value, and the resolver substitutes the caller's default locally on that signal. -flagd's OFREP endpoint answers the same flag with `{"reason": "DISABLED"}` and no `value` and no -`variant` — the same signal in another envelope — and the Python OFREP provider already falls back to -the caller's default for the absent value. It fails these scenarios for a reason unrelated to -architecture, which [its own suite](../../providers/openfeature-provider-ofrep/tests/tck/test_ofrep_conformance.py) -records. The discrepancy belongs upstream rather than papered over here; what it changes locally is -only what a withheld declaration may be read as — not necessarily an impossibility, so read the -adoption's own note for which it was. Withholding still needs no `KnownDeviation`, for the reason -every gated capability does: a deviation records a gap in behaviour the provider is *required* to -have, and this one is optional. - -Nothing in the specification says what a provider owes a disabled flag: -[Requirement 1.4.7](https://github.com/open-feature/spec/blob/main/specification/sections/01-flag-evaluation.md) -is about the SDK propagating whatever reason arrived, and 2.2.5 only lists `DISABLED` among the -reason strings a provider **may** use. So [Appendix F][appendix-f] states the behaviour, the way it -does for `@numeric-coercion`, and gates it. Since spec revision `009afe06` the canonical set carries -four `disabled-*` flags mirroring `boolean-flag`, `string-flag`, `integer-flag` and `float-flag` -exactly, differing only in `state`, and one Scenario Outline of four rows asserts that each resolves -to the caller's default. Each row's default differs from the flag's configured value, so a provider -that ignores the state is caught on the value alone — 2.2.3, a `MUST`. The rows assert neither the -reason, which would rest on 2.2.5's `SHOULD` and its "some other string" — it is pinned in -`gherkin/reason.feature` instead, for a provider that opts into the standard meanings — nor the -variant, since a disabled flag has resolved none: `@disabled-flags` and `@variants` deliberately do -not compose. - -`@standard-reasons` is **a claim, not an exemption**, and it is the one capability whose tag is -carried at the *feature* level. 2.2.5 is a `SHOULD` that goes further than 2.2.4 does: it lets a -provider populate `reason` with one of the listed values *"or some other string indicating the -semantic reason for the returned flag value"*. A provider whose backend reports vendor-specific -reasons is therefore conformant, and asserting an exact reason against it would fail it for something -the specification permits. The suite did exactly that until spec revision `c342461a`, in thirteen -places across `evaluation.feature`, `errors.feature` and `lifecycle.feature`, and it bought very -little: every canonical flag resolves to a value distinct from the caller's default, so a provider -that silently falls back is already caught by the value. - -So the reasons live in `gherkin/reason.feature`, gated as a whole. Declaring the capability is a -provider saying *"I use the standard vocabulary with the standard meanings"*, and that file is what -checks the claim — `STATIC` for a rule-less flag, `TARGETING_MATCH` for a matched rule, `DEFAULT` for -an unmatched one, `DISABLED` for a disabled flag, `ERROR` beside an error code. A provider that does -not declare it **loses nothing**: its values, variants and error codes are asserted everywhere else, -on `MUST` requirements. What the declaration adds is something a report's reader can act on — anyone -building telemetry, dashboards or debugging on `reason` can see that the vocabulary was verified -rather than assumed. `STATIC` for the rule-less rows is the call worth flagging: `types.md` types -`DEFAULT` as *"no dynamic evaluation occurred **or** dynamic evaluation yielded no result"*, so a -provider answering `DEFAULT` there is not defective — it does not use the standard meanings, and -should not declare the tag. - -**Tags compose, and here that is load-bearing.** `TARGETING_MATCH` cannot be observed without -targeting and `DISABLED` cannot be observed unless the backend distinguishes a disabled flag, so two -of the file's scenarios also carry `@targeting` and one also carries `@disabled-flags`. Declaring -`@standard-reasons` alone runs the four `STATIC` rows and the two error scenarios, and skips the -other three with their reason. - -Untagged scenarios are mandatory and always run. `capabilities` defaults to every *declarable* -capability — `DECLARABLE_CAPABILITIES` — and you should narrow it rather than widen it: start from -the default, run the suite, and remove only what your provider genuinely cannot do. - -**What counts as "cannot do" is [Appendix F][appendix-f]'s to say, and there are two questions in it, -asked in order.** First: does your provider owe an answer at all? Where the specification permits -declining — `@numeric-coercion` is defined by no requirement, so a provider that simply does not -coerce is entitled to withhold it — withholding is the honest report however reachable the scenarios -are. Only once a provider *is* attempting the capability does the second question arise, and it is -not about the tag but about its scenarios: declare when at least one scenario gating it can actually -be put to the provider, and withhold only when none can. - -That second rule's two consequences are the ones that bite in practice — a scenario that fails -because the backend serves no fixture for it is not a provider defect and must not be recorded as -one, and a capability withheld for a backend gap is temporary in a way one withheld by choice is -not, so it needs a note saying why or it outlives its reason. The flagd adoption in this repository -decides `@numeric-coercion` and `@large-integers` by it and gets opposite answers; its suite files -cite it rather than restating it, and so should yours. - -Leaving a capability out is the only way to withhold it, and one skip carrying its reason is the -whole mechanism: the scenario's tags say what was asked, the declaration says whether it was -claimed, and the skip says why it was not. - -**The same skip can mean two different things, so say which in your own note.** Both are in this -repository, on one capability: the OFREP adoption withholds `@configuration-change` because nothing -watches the backend — every evaluation is an independent request, and a provider built that way is -not defective — while the in-memory self-test withholds it because the SDK's provider cannot update -its flag set at all, which [Appendix A][appendix-a] requires of it -([python-sdk#620](https://github.com/open-feature/python-sdk/issues/620)). A choice and a defect, -identical in the results, distinguishable only from what the adoption wrote down. - -### A capability this SDK cannot express - -Some capabilities cannot hold in a language *at all* — `@numeric-coercion` where the language has a -single numeric type and "a float requested as an integer" does not name two different requests, -`@large-integers` where the integer accessor is a 32-bit `Integer`. That is a property of the SDK -rather than of the provider, so [Appendix F][appendix-f] makes it the implementation's job: -`INEXPRESSIBLE_CAPABILITIES` lists them, `TckConfig` refuses to let you declare one, and the error -names the property of the SDK that puts the question out of reach. You are not expected to know this -about your language, and three suites each remembering it separately is three chances to put a claim -in a report that no scenario could have verified. - -**`INEXPRESSIBLE_CAPABILITIES` is empty in Python, and that was measured rather than assumed.** -`int` is arbitrary-precision, and `get_integer_details` and `get_float_details` are separate -accessors reaching separate provider methods and type-checked against `int` and `float` separately — -so all four questions the two tags ask can be put, and all four were asked and answered. Both are -ordinary declarable capabilities here. - -A provider that gets one of them *wrong* is a different thing and does not belong here. Measured by -declaring `@numeric-coercion` in both flagd suites and running it: flagd's in-process resolver -refuses `0.5` as an integer and widens `10` to a float, and its RPC resolver widens `10` and -silently narrows `0.5` to `0`, which is the one thing the lossy scenario forbids. Two different -answers to the same three questions, from two resolvers of one provider — which is what a language -that *could not ask* them would make impossible. That split is a defect in one resolver of one -implementation, and **both** resolvers declare the tag: the one that narrows carries a -`KnownDeviation` and the one that does not carries none, which is the shape [Appendix F][appendix-f] -prefers and the opposite of what this paragraph used to prescribe. Neither has anything to do with -Python. (The third scenario fails on both for a third reason again: flagd-testbed seeds no -`integral-float-flag` at all — the backend's gap, not the provider's, and recorded as such rather -than as a reason to withhold.) - -**A reservation and an inexpressibility are not the same refusal**, and the messages and the skip -reasons deliberately differ: - -| | reserved (`@caching`) | inexpressible | -|---|---|---| -| Why | no scenario anywhere carries the tag | the scenarios exist and this SDK cannot ask them | -| Scope | every language | one language | -| Lifetime | expires when the specification adds scenarios | permanent, until the SDK changes | -| The skip says | the capability has no scenarios yet | no provider in this language can be asked | - -Anyone reading a report has to be able to tell *"this provider declined"* from *"no provider in this -language can be asked"*, because only the first says anything about the provider. - -A reserved capability is documented so the vocabulary has a place for it once scenarios exist, and -until then it **must not be declared**. Nothing carries the tag, so declaring it cannot be verified, -cannot produce a skip, and tells anyone reading the declaration only that something was claimed and -nothing examined. `TckConfig` raises if you name one in `capabilities`, and -`DECLARABLE_CAPABILITIES` excludes them — which is the case that matters, because "every capability -except X" is how a reserved tag gets declared by accident rather than by decision. One -implementation's published conformance report asserts `@targeting` and `@caching` for exactly that -reason, back when both were reserved. - -`@caching` is the only reserved tag left. Leaving one reserved once it *has* scenarios would be the -mirror of the mistake the set exists to prevent — a capability that can be verified, refused the -chance — so `@targeting` moved out of it the moment the specification gave it three. - -**A scenario carrying a reserved tag fails the run.** `TckConfig` refuses to let anyone declare a -reserved capability, so the gate skips every scenario carrying one — for a capability nobody is -permitted to claim, which leaves a gap in the report that the provider may not have. Appendix F calls -that the unclaimable capability, and nothing else notices it: the run is green and the report is -well-formed. So the plugin refuses to continue, naming the tags. - -Two mistakes end there and the message names both remedies. Either the tag arrived with the canonical -feature files, because the specification wrote the scenarios the reservation was held open for and -this package has not followed — take the tag out of `RESERVED_CAPABILITIES` and decide, per adoption, -whether to declare it. Or it arrived from a feature file of your own under `extensions/`, in which -case pick a tag of your own: a reserved tag gates nothing and can never be declared, so a scenario -carrying one can never run. - -The two halves are read differently, and that is deliberate. The canonical tags come from the -packaged files rather than from the collected run, so a `-k` or `--deselect` cannot narrow a run past -the specification's half; your extensions have no such source — the directory is found from your test -module — so those tags come from what was collected. Both are the parsed Gherkin tags, never the file -text: `gherkin/events.feature` names `@caching` in a `#` comment saying where those scenarios will go -once they exist, and a text scan would fail every adoption over a sentence. - -`@numeric-coercion` deserves a note, because it is the one capability here that **the specification -does not define**. OpenFeature has a single numeric type on purpose — `number` is "a numeric value of -unspecified type or size", and languages **may** differentiate between integers and floats "as idioms -dictate" — so no requirement says what a provider must do when a value does not fit the accessor it -was asked through. That gap is [open-feature/spec#430](https://github.com/open-feature/spec/issues/430). - -The rule this tag is tested against is therefore **borrowed, not normative**: lossless coercion is -permitted, lossy coercion must fail — `10.0` requested as an integer must succeed, `0.5` must not. -It comes from flagd's -[numeric coercion ADR](https://github.com/open-feature/flagd/blob/main/docs/architecture-decisions/numeric-coercion.md), -which is scoped to flagd's own implementations, and the tag carries that name — it was -`@strict-numeric-typing` — because two vocabularies for one observable property is worse than one -borrowed name. **A provider that behaves differently is not violating the specification** — but that -does not leave a missing declaration free to interpret, and [Appendix F][appendix-f]'s note on this -tag says which is which. A provider that *attempts* the coercion and gets one direction wrong -declares the capability, lets the lossy scenario fail and records a `KnownDeviation` beside it, -because "it coerces, and one direction is wrong" is what a skip cannot say. Withholding is for a -provider that *cannot attempt* it: a language with a single numeric type, or one that hands every -variant back untouched and never coerces — which is what this SDK's `InMemoryProvider` does, and why -neither in-memory self-test declares the tag. - -Both halves are tested, and a provider declaring the tag must satisfy all three scenarios: `float-flag` -(`0.5`) requested as an integer is a `TYPE_MISMATCH`; `integral-float-flag` (`10.0`) requested as an -integer is `10`; `integer-flag` (`10`) requested as a float is `10.0`. Rejecting every float is an easy -way to pass the first, and the other two are what stop it. - -The width of the integer accessor is the related property, and it is a capability of its own because -it belongs to the SDK rather than to the provider. Every language can ask for 2^31 − 1, so that -precision scenario is untagged; only the one asking for 2^53 − 1 carries `@large-integers`. A Python -`int` is unbounded, so a Python provider declares it unless something of its own — a 32-bit field in -its wire format, a float on the way through — narrows the value. - -### Steps that reach the provider directly - -Everything the suite asks of a provider goes through an OpenFeature client, as an application's -would — except three steps. `the provider is shut down` and `the provider is initialized again` call -the provider's own `shutdown()` and `initialize()` on the registered instance, and -`the provider metadata name should not be empty` asks it for `get_metadata()`. Going through the SDK -would test the registry's bookkeeping as much as the provider, and Appendix B already does that; it -would also make a double shutdown impossible to express, since the registry calls `shutdown` once -per registration. - -The registry is not told. The client keeps pointing at the same instance, so an evaluation after -re-initialising reaches the very object that was shut down and brought back. When the scenario ends, -the SDK shuts the provider down once more on its own — requirement 2.5.3 makes that second call -harmless, and the suite relies on it. A direct call that outlasts `TckConfig.ready_timeout` is given -up on and fails its scenario with a message rather than hanging the session. - -### Declaring more than a capability set - -One further field on `TckConfig` says something a capability set cannot, and it is a declaration -rather than a switch: it changes neither which scenarios run nor what they assert. - -`known_deviations` acknowledges a gap against something the specification does *not* treat as -optional. It is an acknowledgement and not an excuse: the scenario still fails and the suite still -fails with it. What the declaration adds is that the gap was known rather than a surprise. +Untagged scenarios are mandatory and always run. `capabilities` defaults to +`DECLARABLE_CAPABILITIES`; narrow it rather than widen it. Tags compose, so declaring +`@reinitialization` without the `@lifecycle` its feature carries leaves that scenario skipped. +**What counts as "cannot do" is [Appendix F][appendix-f]'s rules for declaring**; the adoptions here +cite them rather than restating them, and so should yours. + +`TckConfig` refuses two declarations at construction rather than letting them reach a report: a +**reserved** capability, which no scenario carries, and one this language's SDK **cannot express** — +the two errors and the two skip reasons deliberately differ, and a scenario arriving with a reserved +tag fails the run. `INEXPRESSIBLE_CAPABILITIES` is **empty in Python**, measured rather than assumed: +`int` is arbitrary-precision and the integer and float accessors reach separate provider methods, so +both tags are ordinary declarable capabilities here. + +### Known deviations + +`known_deviations` says what a capability set cannot: that the provider fails something it is +**required** to do. It is a declaration, not a switch — the scenario still runs and still fails. ```python TckConfig( @@ -531,365 +200,127 @@ TckConfig( ) ``` -- **`summary` is required.** A deviation with no summary records that something is wrong without - saying what, which leaves a reader worse off than the bare skip or failure it accompanies. -- **`issue` is optional**, and `KnownDeviation.untracked(summary=...)` is the form for a gap that is - not tracked anywhere yet. Naming an untracked defect is still what separates it from a capability - the provider chose to withhold; prefer the tracked form as soon as there is somewhere to point. -- **`capability` is optional**, and left out when the gap is against a mandatory, ungated scenario. - A reserved capability is refused: no scenario carries the tag, so there is nothing to deviate - from. - -It is legitimate in two shapes, and **prefer the first**: +`summary` is required; `issue` is optional, with `KnownDeviation.untracked(summary=...)` for a gap +tracked nowhere yet; `capability` is optional, left out for a mandatory ungated scenario, and a +reserved one is refused since no scenario carries the tag. [Appendix F][appendix-f] settles which of +the two shapes — declare and let it fail, or withhold and let it skip — to reach for. -1. **The capability is declared, the scenario runs, and it fails.** The failure stays visible and - the deviation says it is known and why. -2. **The capability is withheld and its scenarios skip.** Legitimate only when the provider cannot - attempt the behaviour at all, so running the scenario would establish nothing. The deviation then - explains the absence, so a reader can tell a defect from a design decision. +## Running it -Withdrawing a capability *in order to* turn a failing scenario into a skip is the failure mode this -field exists to prevent. Where the specification permits the choice, withholding the capability -**is** the honest report and a deviation entry would assert a defect that does not exist. - -## Controlling the backend - -`BackendControl` is the single seam between the scenarios and whatever manipulates the backend. Step -definitions never talk to a backend directly, which is why the same Gherkin runs unchanged against a -containerised backend and against a provider manipulated in-process. - -**If your provider talks to a backend, drive it over the HTTP control API** — the document is -available as `control_api_spec()`, and `HttpControl` is the client for it. That API is the normative -contract for those providers, and it is what makes a conformance claim portable: another language's -TCK drives the same endpoints against the same stack and must get the same answers. +**Keep the adoption suite out of the default build, give it a task of its own, and write down that +you did** — the reasoning is Appendix F's ["Running the suite in CI"][appendix-f]. The Python part is +these tasks, which CI reaches through `poe cov`: -```python -control = HttpControl(f"http://localhost:{container.get_launchpad_port()}") +```toml +[tool.poe.tasks] +test = ["test-default", "test-tck-collect"] +test-cov = ["test-cov-default", "test-tck-collect"] +test-default = "pytest tests --ignore=tests/tck" +test-cov-default = "coverage run -m pytest tests --ignore=tests/tck" +test-tck = "pytest tests/tck" +test-tck-collect = "pytest tests/tck --collect-only -q" ``` -`HttpControl` is built on `urllib.request` alone, so the TCK gains no HTTP client. You do not build -one yourself when you use the Compose harness below: it is handed to you as `tck_backend.control`, -already awaited ready. - -**A control must say which path it drove the backend through.** `control_api` is a required member of -`BackendControl`, typed `ControlApi` — `Literal["http", "in-process"]` — with no default and no -inference from the control's concrete type. `HttpControl` answers `"http"`, `InProcessControl` -answers `"in-process"`, and a custom control states its own. It is the one fact that decides what -everything else in a report is worth: the same scenarios passing over the control API and passing -through in-process manipulation of a provider that *does* have a backend are not the same claim, and -this is the only field that separates them. Nothing outside a control can tell the two apart, and -every run is one or the other — so an absent value would not be "no claim made" but an unfalsifiable -one. The type is closed, so `"HTTP"` or `"grpc"` is a type error here rather than a conformance -report that fails schema validation somewhere else. +Put a comment above them saying why, so the exclusion cannot read as an oversight, and record the +current tally where a reviewer will see it — **read the tally rather than the green check**, since a +conformance suite carries failures by design. Both adoptions here do exactly that. -### The container stack - -The suite starts it. An adopter used to write the container wrapper — and every adopter wrote the -same one, which is why the flagd adoption alone carried a 122-line `conftest.py` and a 170-line -`suite.py` of it. `ComposeBackend` is the whole declaration: +Two Python-specific notes on top of the appendix: -| field | required | default | meaning | -| --- | --- | --- | --- | -| `compose_file` | yes | — | path to the Compose file, resolved relative to the package directory | -| `backend_ports` | yes | — | container-internal ports the **provider** connects to. The control port is exposed automatically and must not be listed here | -| `backend_service` | no | `"backend"` | the Compose service hosting both the control API and the backend | -| `control_port` | no | `8080` | container-internal port of the control API | -| `additional_ports` | no | `{}` | extra service to ports, for a stack with more than one service. Resolved through the endpoint by service name | -| `backend_configuration` | no | `"default"` | the configuration name passed to `POST /start` | -| `startup_timeout` | no | `60.0` | seconds to wait for the stack and its control API to become reachable | +- **Docker is not what decides it here.** `tests/e2e` needs Docker too, has needed it for years, and + still runs in the default build on `ubuntu-latest`. The exclusion rests entirely on the second + reason, that a conformance suite's honest output is red. +- **`--ignore` does not import the suite, so nothing checks that it still would**, and `mypy` here is + configured over `src` alone. Hence `test-tck-collect` in both default tasks: it imports every test + module and resolves the feature files while starting no container, which is the appendix's "keep it + compiling" in the form Python has available. -Those names and defaults are fixed across all four languages' TCKs, so a provider shipped in two of -them writes one Compose file and two declarations against it. +## Extending it -`tck_backend` is a session-scoped fixture this package's plugin supplies, and it yields two things: - -- `tck_backend.control` — the `HttpControl`, already awaited ready. Hand it to `TckConfig.control`. - One per stack: it remembers whether a disconnect left the backend down, so two suites driving the - same backend must share it. -- `tck_backend.endpoint` — `host`, `port(internal)` and `port(internal, service=...)`. This is a - **factory argument, not a field**: the mapped ports do not exist until the stack is up, which is - why `TckConfig.new_provider` is a factory called once per scenario. - -Your Compose file must **not pin host ports**. Docker assigns them dynamically and the harness -discovers them after startup; a pinned host port makes the suite unrunnable in parallel and collides -with whatever you already have listening. Declaring `backend_ports` is what lets the harness say -"the Compose file does not publish 8013" at startup rather than leaving you with a provider that -cannot connect three scenarios later. - -Startup is a real readiness check rather than a pause: the stack comes up with -`docker compose up --wait`, then every declared port is waited on until it accepts a connection, -then `HttpControl.await_ready()` probes `GET /healthz` until the control API answers. There is -deliberately **no settle after a control call**. Java had a fixed 50ms one, and `control-api.yaml` -now states what makes it the wrong instrument: every state-changing endpoint — `/start`, `/change`, -`/reset` — must not return until the new state is actually being served, so a delay here covers a -window the backend is specified to close, and a suite that sleeps instead of holding the API to that -promise stops being able to detect when the promise breaks. The delay is also un-tunable, because -the window is a property of the backend and not of the harness. - -Backends do still break it — flagd-testbed's launchpad returns from `/start` as soon as `/readyz` -answers, which is roughly 40 ms before the flags are evaluable, and -[flagd-testbed#394](https://github.com/open-feature/flagd-testbed/pull/394) is open and unmerged. A -provider that blocks in `initialize` absorbs that window; a stateless one lands in it. Where you are -stuck with such a backend the wait belongs in **your adoption**, set explicitly and citing the -defect, so it reads as a named workaround for one backend and disappears when the backend is fixed — -see the OFREP adoption's `SettledControl`. It does not belong here, where every future adopter would -inherit it without knowing why. - -`testcontainers` is an **optional** extra rather than a dependency: +Provider behaviour the specification does not describe — flagd's `fractional` targeting, a vendor's +own rollout rule — belongs in the same run rather than a second harness. Create an `extensions` +directory beside the module that calls `scenarios()`, with step definitions in a `conftest.py`: ``` -pip install 'openfeature-tck[compose]' +tests/ +├── conftest.py # your step definitions +├── test_conformance.py # the fixture and the one call, unchanged +└── extensions/ + └── fractional.feature ``` -An in-memory adopter should not have to install container tooling to run a suite that never starts a -container, so `compose.py` imports it lazily and says so if it is missing. - -If you want the fixture under a different name or scope, `run_compose_backend()` is the generator -behind it: - ```python -@pytest.fixture(scope="session") -def tck_backend(): - yield from run_compose_backend(ComposeBackend(...)) -``` - -Two of its behaviours are worth knowing about: - -- **`/reset` is optional and the fallback is automatic.** `prepare_scenario()` prefers `POST /reset`, - which restores the flag baseline with no availability blip; a backend without it answers 404 or - 501 and the client falls back to `POST /start?config=default`. The probe happens once per suite. - flagd-testbed's launchpad registers only `/start`, `/restart`, `/stop` and `/change`, so that - fallback is the normal path today. -- **After a disconnect it starts rather than resets.** `/reset` restores flag *state*; it is not - specified to bring a stopped backend back up. +# conftest.py +from pytest_bdd import then -Two of the API's requirements are easy to get wrong: - -- **Containers are never stopped or restarted mid-suite.** Unavailability is simulated *inside* the - running stack. Container orchestrators assign host ports dynamically and cannot reliably preserve - them across a restart, so restarting silently invalidates every provider already pointed at the - old port, and the failure looks like a flaky provider. -- **An outage must be observable as a change in availability, never as a change in flag values.** - `reconnect()` is `POST /start` with the configuration already in effect, which restores the same - baseline. -- **There is no binding for `POST /restart`.** It simulates a *bounded* outage and is `[OPTIONAL]` in - `control-api.yaml`, because no shipped scenario reaches it: the disconnect/reconnect scenario is - written as an unbounded outage — "the connection is lost", then "the connection is restored" — - which is `disconnect()` then `reconnect()`, so the scenario ends the outage when it is ready rather - than guessing in advance how long the provider needs to notice one. What would bring the endpoint - back is a `@caching` scenario asserting what a stale provider serves *during* an outage, which - needs the flag-state preservation `/restart` has and `/stop` + `/start` does not. - -### Providers with no backend - -An in-memory, environment-variable or file-based provider has nothing to connect to. Those may -control the backend in-process, where flag operations are direct manipulations of the provider's own -state. `InProcessControl` is the reference. - -This is a narrow allowance and the obvious thing to abuse. **A provider with an external backend -must use the control API.** Reaching into an external backend from inside the test process — a -test-only admin client, a shared database handle, a hook inside the provider — produces a suite that -passes while proving nothing, because the path it exercised is not the path the contract describes. - -Connection-dependent scenarios have no meaning without a connection, so a backend-less control -simply does not implement `ConnectionControl`, leaves `STALE` and `UNAVAILABLE_INIT` undeclared, and -those scenarios are skipped with their reason. - -Such a control reports `control_api` as `"in-process"`, and that is the whole reason the field is -required rather than guessed: the allowance is only narrow if a report says when it was taken. - -## Findings - -Four, all confirmed by running the suite rather than by reading code. - -### 1. A boolean satisfies an Integer request - -`boolean-flag` evaluated through `get_integer_details` returns `True` with reason `STATIC` and **no -error code**, where the specification requires the code default and `TYPE_MISMATCH`. The client -type-checks with `isinstance(value, int)`, and `bool` is a subclass of `int` in Python. - -This is **Python-specific** — the identical scenario passes in every other language's suite, which -is a fair advertisement for having more than one implementation. Tracked as -[open-feature/python-sdk#619](https://github.com/open-feature/python-sdk/issues/619). - -The self-test marks that one row `xfail(strict=True)` with a pointer to the issue, so it stays -visible in the report and un-hides itself automatically once the SDK is fixed. - -### 2. The in-memory provider cannot update its flag set - -[Appendix A][appendix-a] requires an SDK's in-memory provider to support updating the flag set and -emitting `PROVIDER_CONFIGURATION_CHANGED`. Python's copies its mapping in the constructor and -exposes nothing to change it. Tracked as -[open-feature/python-sdk#620](https://github.com/open-feature/python-sdk/issues/620). - -Only half the machinery is missing — `AbstractProvider` already supplies -`emit_provider_configuration_changed` — which is why `ControllableInMemoryProvider` here is a small -subclass rather than a reimplementation, and why it should port back to the SDK as a method. - -`test_in_memory_conformance` therefore withholds `CONFIGURATION_CHANGE` for a defect, which again is -the [Appendix F][appendix-f] self-test carve-out rather than something an adoption may copy. It meets -the condition differently from finding 4: the scenarios are not only skipped, they are *run* — by -`test_controllable_conformance`, against the subclass that supplies what the SDK lacks, so the step -definitions and the change path stay covered and the gap is written down in `PlainMemoryControl`'s -`change_flag`, which raises rather than pretends. The one thing this pin does not do is go red on -its own when the SDK is fixed; nothing fails at that point, the subclass just becomes redundant. - -### 3. The in-memory provider does not coerce numbers - -`integral-float-flag` (`10.0`) requested as an integer returns the code default with `TYPE_MISMATCH`, -and `integer-flag` (`10`) requested as a float does the same. The provider hands each variant back -untouched and the client's type check is `isinstance`-based, so neither lossless direction happens. -The lossy scenario passes — every float is rejected — which is exactly the shortcut the two lossless -scenarios exist to catch. - -This is not a defect: `@numeric-coercion` is optional, and the specification does not define the -behaviour. So neither in-memory self-test declares the tag, and the three scenarios are skipped with -that reason rather than failing. - -### 4. The in-memory provider ignores a flag's state - -`InMemoryFlag` has a `State` enum with an `ENABLED` and a `DISABLED` member, takes one in its -constructor, and **never reads it**: `InMemoryFlag.resolve` returns the default variant's value with -reason `STATIC` whatever the state, and `InMemoryProvider._resolve` looks only for a missing key. So -all four `disabled-*` flags are served exactly as their enabled counterparts are. - -`canonical_flag_set` is not where this stops. `_decode_canonical_flags` reads the canonical file's -`"state": "DISABLED"`, validates it against `InMemoryFlag.State` and passes it through faithfully — -the self-tests pin that it reaches exactly those four flags and no others. The state survives -decoding and then has no effect. - -Measured before the tag was gated: all four rows failed on the value in both in-memory suites, with -`disabled-boolean-flag` resolving to `True` against a caller default of `false`. So neither suite -declares `@disabled-flags` and the four scenarios are skipped with that reason. - -Unlike finding 3 this is a field the SDK offers and does not honour, so it is a defect rather than a -choice — and withholding a capability for a defect is the thing [Appendix F][appendix-f] tells an -*adoption* not to do. What licenses it here is the appendix's self-test carve-out: these two suites -are a fixture for the harness rather than a report about a third party, they run in the ordinary -build where a permanently failing scenario is a broken build rather than a finding, and the fix is an -SDK release away. **The carve-out's condition is that the defect is pinned by a test of its own, and -it is**: `test_every_packaged_flag_resolves_to_its_packaged_default_variant` sweeps the four -`disabled-*` flags with everything else and asserts that each resolves to its own default variant — -so the behaviour is asserted rather than merely skipped, and that test turns red the day the SDK -starts honouring `DISABLED`, which is when the capability becomes declarable here. It is not filed -against the SDK yet. - -## Where the assets come from +from openfeature.contrib.tools.tck import TckState -The Gherkin feature files, the canonical flag set and the control-API document are **not owned by -this repository**. They are the language-agnostic conformance artifacts defined in -[open-feature/spec][spec] under `specification/assets/provider-tck/`, and every language's TCK ships -the same ones — which is the only reason a conformance claim means the same thing in Python as it -does in Java. -**Adopting this package needs no submodule.** The assets are copied into the wheel and the sdist at -build time, so `pip install openfeature-tck` gives you everything the suite runs on. +@then("the fractional rule splits the population") +def fractional_splits(tck_state: TckState) -> None: ... +``` -**Contributing to this package does.** The spec is a git submodule at -`tools/openfeature-tck/spec`, and the copies under -`src/openfeature/contrib/tools/tck/` are gitignored and generated: +**No registration, no option and no new argument.** pytest collects `conftest.py` and pytest-bdd +resolves steps through the fixture system, so the canonical vocabulary is in scope beside your own, +and `tck_state` is the per-scenario state those steps use — your scenario runs against the provider +the suite registered, in the same lifecycle and reset. `feature_paths()` already includes an +`extensions` directory when there is one, so the `scenarios()` call never changes. + +**Your scenarios cannot stand in for ours.** Canonical features are identified by their path under +`gherkin/` and yours under `extensions/`, and `extensions.py` refuses a file of yours under the +reserved prefix, or two that would share one uri. Both are reachable here, because pytest-bdd names a +feature file by its parent directory joined to its own name — so `extensions/gherkin/errors.feature` +would land on the canonical `errors.feature`'s uri, and a run could go green having asked the +adopter's questions instead of the specification's. + +## Python-specific notes + +**A boolean satisfies an Integer request**, because the client type-checks with +`isinstance(value, int)` and `bool` subclasses `int`. `boolean-flag` through `get_integer_details` +returns `True` with no error code where the specification requires the code default and +`TYPE_MISMATCH`, and the identical scenario passes in every other language's suite. Expect it in your +own run until a release carries [python-sdk#619](https://github.com/open-feature/python-sdk/issues/619); +the same cause reaches flagd-core's type check +([python-sdk-contrib#417](https://github.com/open-feature/python-sdk-contrib/issues/417)). + +**Three steps reach the provider directly rather than through a client**, because going through the +SDK would test the registry's bookkeeping as much as the provider: the shutdown and re-initialise +steps call the registered instance's own `shutdown()` and `initialize()`, and the metadata step asks +it for `get_metadata()`. The registry is not told, so the SDK shuts the provider down once more when +the scenario ends — requirement 2.5.3 makes that harmless — and a direct call outlasting +`ready_timeout` fails its scenario rather than hanging the session. + +**The SDK's `InMemoryProvider` is why this package ships `ControllableInMemoryProvider`** and why the +self-tests declare less than they otherwise would: it cannot update its flag set, which +[Appendix A][appendix-a] requires +([python-sdk#620](https://github.com/open-feature/python-sdk/issues/620)), never reads +`InMemoryFlag.state` ([python-sdk#627](https://github.com/open-feature/python-sdk/issues/627)), and +hands each variant back untouched, so it does not attempt numeric coercion at all — a permitted +choice rather than a defect, and the reason `@numeric-coercion` is simply not declared there. + +## Contributing + +The Gherkin, the canonical flag set and the control-API document are **not owned by this +repository** — they are the artifacts in [open-feature/spec][spec], copied into the wheel and the +sdist at build time. So adopting needs no submodule and contributing does: ```bash git submodule update --init tools/openfeature-tck/spec -poe test # runs `poe sync-spec-assets` first -``` - -The copies carry a `DO-NOT-EDIT.txt` because editing them forks the definition of conformance, which -is the one thing this suite exists to prevent. A change goes to [open-feature/spec][spec] first; -then bump the submodule pin here. Committing no copies means the spec revision this package targets -is recorded by the pin and nowhere else, so the two cannot drift apart unnoticed. - -### The checkout is part of the sync, not something you remember - -**A rebase moves the gitlink and not the submodule's working tree.** So a checkout can have a pin -naming one revision and assets on disk from another, with nothing in the build saying so. That is -not hypothetical: a sync after a rebase here copied the previous pin's Gherkin over the capability -the suite had just been given, and the only thing that noticed was a self-test comparing the -capability enum against the assets. That guard fires for one symptom. A pin that changes nothing but -the *content* of a scenario would pass every guard in this package and still run the wrong suite — -which is exactly what happened in another language, where an entire adoption suite ran against stale -assets and reported byte-identical numbers to the run before it. - -So `poe sync-spec-assets` brings the submodule to the pinned revision itself before it copies -anything, and `poe test` depends on the sync. **The suite cannot run against assets it did not just -check out**, and `git submodule update` is no longer something an operator has to remember after a -rebase. If the pinned commit is not reachable, nothing is copied and the build stops with a message -saying so, rather than quietly testing the wrong questions. - -**Which guarantee this is, exactly.** Not "a stale copy is impossible": these assets reach the -package by being *copied*, so a copy can always be made wrong — by hand, or by a sync that never -ran. What the wiring buys is narrower and worth stating in its own words — **the suite cannot run -without a fresh sync, and a sync cannot succeed against any revision but the pinned one.** Go's TCK -has the stronger property without doing anything, because it consumes the assets as a nested Go -module out of a read-only, checksum-verified module cache: there is no second artifact to go stale -and the only way past it is a visible `replace` line. The two fail differently, so it is worth -knowing which one you have. - -Two escape hatches, both loud: - -- `OPENFEATURE_TCK_SPEC_UNPINNED=1` copies whatever is checked out in `spec/`, for drafting a change - to the canonical assets before there is a revision to pin. It warns on every sync and names the - revision it used. -- Where the pin cannot be read at all, the sync warns and continues. An unpacked sdist is the - ordinary case — no repository, no pin, and the assets are already in the tree. The other is a - linked git worktree whose `.git` file names a path outside the running process's filesystem - namespace, such as a Windows worktree driven from WSL: git answers inside the submodule and not in - the superproject. The guarantee is genuinely off there, which is what the warning says; run the - sync from a shell that can see the superproject. - -This mirrors what `openfeature-flagd-api-testkit` already does for the flagd test harness. - -## The self-tests - -| Suite | Subject | Why | -| --- | --- | --- | -| `test_in_memory_conformance` | the SDK's `InMemoryProvider` | reference adoption for a backend-less provider | -| `test_controllable_conformance` | `ControllableInMemoryProvider` | the only suite that exercises the configuration-change path — see finding 2 | -| `test_in_process_control` | `InProcessControl` and the canonical flag set | pins what the Gherkin cannot assert about itself, including that the in-memory flag set is decoded from `canonical-flags.json` — every flag served under the file's own default variant, with the Python type the file wrote, and `state` reaching exactly the four `disabled-*` flags | -| `test_lifecycle_steps` | the steps that call the provider directly | the in-memory suites skip `@lifecycle`, so the shutdown, re-initialise and metadata steps are driven against a recording provider instead | -| `test_declaration` | what a `TckConfig` claims | none of it is observable in a pass or a fail, so nothing else would catch it | -| `test_extensions` | an adopter's own scenarios | an extension runs inside the canonical suite, changes nothing for an adopter who has none, and cannot take a canonical scenario's identity | -| `test_http_control` | `HttpControl` | the `/reset` fallback, the disconnect bookkeeping, the control API it reports and the absence of a `/restart` binding, against a stubbed control API | -| `test_spec_assets` | where the conformance assets came from | the submodule checkout and the copy are the only thing standing between a run and the wrong questions, and a stale copy is invisible in a pass or a fail | - -``` -221 passed, 42 skipped, 2 xfailed +poe test # syncs the assets first; 221 passed, 42 skipped, 2 xfailed, no Docker ``` -No Docker and no network beyond loopback. The conformance suites take under a second; -`test_extensions` takes most of the rest, because the properties it checks are properties of a whole -pytest session and it runs a generated adoption in a subprocess to check them. - -Neither in-memory suite declares `@lifecycle`, so the six lifecycle scenarios — three about -initialisation, three about shutdown — are skipped in both. That is the point: with no backend to -reach, the initialisation ones would pass without testing anything — which is what they did while the -feature was gated on `@events`. Neither declares `@numeric-coercion` either, for the reason in -finding 3, so its three scenarios are skipped too. Neither declares `@targeting`: both resolve the -same decoded flag set, and `canonical_flag_set` deliberately ignores `targeting-key-flag`'s rule -rather than becoming a second implementation of somebody else's evaluator, so those three scenarios -are skipped as well. Neither declares `@disabled-flags` either, for the reason in finding 4 — the -state reaches the flag set and the SDK's provider never reads it — so its four rows are skipped in -both. Both declare `@variants`, since an in-memory flag set is keyed by variant name. - -Both declare `@standard-reasons`, and it was measured before it was declared: `InMemoryFlag.resolve` -reports `Reason.STATIC` for every flag in the decoded set, and a missing flag and a type mismatch -both arrive with reason `ERROR` beside their error code, so the four rule-less rows and the two error -scenarios pass in each suite. The remaining three scenarios in `reason.feature` compose the tag with -`@targeting` and `@disabled-flags`, neither of which is declared, so they are skipped in both — which -is the composition working rather than a gap, since a reason cannot be observed without the behaviour -that produces it. - -## Known gaps - -- **Evaluation context passthrough is verified only for the targeting key.** `targeting-key-flag` - resolves differently for a matching context, so the `@targeting` scenarios catch a provider that - drops the context — no echo operation needed for that. What is still unverified is that the - *whole* context arrives intact: a provider that forwards the targeting key and silently discards - every other attribute passes. That needs either an echo operation on the control API or a second - canonical flag whose rule keys on a custom attribute. -- **Caching, hooks and flag metadata** are not covered. Appendix F's ["Known gaps"][appendix-f] is - the list of record, and it now also carries the constraint a `@caching` scenario has to be written - against. +The copies under `src/` are gitignored, generated and carry a `DO-NOT-EDIT.txt`: a change goes to +[open-feature/spec][spec] first, then the submodule pin moves here, and that pin is the only record +of the revision this package targets. **The checkout is part of the sync, not something you +remember** — a rebase moves the gitlink and not the submodule's working tree, which is how an +adoption suite in another language ran a whole pass against the previous pin's feature files and +reported numbers identical to the run before it. So the sync checks the submodule out at the pinned +revision before copying and `poe test` depends on it: **the suite cannot run without a fresh sync, +and a sync cannot succeed against any revision but the pinned one.** +`OPENFEATURE_TCK_SPEC_UNPINNED=1` opts out while drafting a change to the assets; where the pin +cannot be read at all the sync warns and continues with the guarantee off, which covers an unpacked +sdist and a worktree whose `.git` points outside the running process's filesystem namespace, such as +a Windows worktree driven from WSL. [appendix-a]: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md [appendix-f]: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md From 707dd6d39cf0bf76bbc8990a2e670b46fa60c899 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 22:33:54 +0200 Subject: [PATCH 44/46] docs(tck): lay the extension example out where the tasks actually look The extension tree was rooted at `tests/`, while the task block ten lines above it excludes `tests/tck` and `poe test-tck` runs that path. An adopter following both sections got a conformance module the exclusion does not reach, which is the documented-command-that-does-not-work failure this effort started from. The example module is renamed with it. `test_conformance.py` inside `tests/tck` says conformance twice, and the module name selects nothing -- the directory does -- so the example now shows what both adoptions here do, with a sentence saying why. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index a54c36c62..41b8a0a86 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -242,13 +242,17 @@ own rollout rule — belongs in the same run rather than a second harness. Creat directory beside the module that calls `scenarios()`, with step definitions in a `conftest.py`: ``` -tests/ +tests/tck/ ├── conftest.py # your step definitions -├── test_conformance.py # the fixture and the one call, unchanged +├── test_my_provider.py # the fixtures and the one call, unchanged └── extensions/ └── fractional.feature ``` +The directory is `tests/tck` because that is what the tasks above exclude and what `poe test-tck` +runs; the module inside it needs no `conformance` or `tck` in its name, since the directory is what +selects the suite. Both adoptions in this repository are laid out that way. + ```python # conftest.py from pytest_bdd import then From 01bb657f243c94c20e7bfef9e88c1794e9fc480a Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 23:04:24 +0200 Subject: [PATCH 45/46] docs(tck): say what the collect step guarantees, and what keeps it reachable Two things the README asserted without saying why they hold. It cannot pass vacuously: pytest exits 5 when a path collects nothing and 4 when the path does not exist, so a suite that moved out from under the task fails the step instead of skipping it. That is the property the other languages are being pointed at this mechanism for, and it was worth writing down rather than leaving a reader to trust it. And it only holds if the step runs. poe aborts a sequence at its first failing subtask, so the compile check sat behind the default suite's result -- which on the flagd package is red on this branch stack, so the check had in fact never run there. The task block gains `ignore_fail = "return_non_zero"`, and both adoptions' `pyproject.toml` gain it with it. Signed-off-by: Simon Schrottner --- tools/openfeature-tck/README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/tools/openfeature-tck/README.md b/tools/openfeature-tck/README.md index 41b8a0a86..589370d3d 100644 --- a/tools/openfeature-tck/README.md +++ b/tools/openfeature-tck/README.md @@ -213,8 +213,8 @@ these tasks, which CI reaches through `poe cov`: ```toml [tool.poe.tasks] -test = ["test-default", "test-tck-collect"] -test-cov = ["test-cov-default", "test-tck-collect"] +test = { sequence = ["test-default", "test-tck-collect"], ignore_fail = "return_non_zero" } +test-cov = { sequence = ["test-cov-default", "test-tck-collect"], ignore_fail = "return_non_zero" } test-default = "pytest tests --ignore=tests/tck" test-cov-default = "coverage run -m pytest tests --ignore=tests/tck" test-tck = "pytest tests/tck" @@ -233,7 +233,11 @@ Two Python-specific notes on top of the appendix: - **`--ignore` does not import the suite, so nothing checks that it still would**, and `mypy` here is configured over `src` alone. Hence `test-tck-collect` in both default tasks: it imports every test module and resolves the feature files while starting no container, which is the appendix's "keep it - compiling" in the form Python has available. + compiling" in the form Python has available. It cannot pass vacuously — pytest exits 5 on a + directory that collects nothing and 4 on a path that does not exist, so a suite that moved out from + under the task fails it rather than skipping it. `ignore_fail = "return_non_zero"` is what keeps the + check reachable: poe otherwise aborts the sequence at the first failing subtask, and a compile check + that only runs while the rest of the build is green is not a check. ## Extending it From 635ac58a511be1732f787e34bb00f698c44f536c Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 08:24:15 +0200 Subject: [PATCH 46/46] docs(tck): let Appendix F own the capability vocabulary capability.py was 537 lines of prose to 52 of code, and most of the prose was Appendix F restated: the @standard-reasons section almost paragraph for paragraph, the numeric-coercion note, how @reinitialization came to be gated, the reserved-versus-inexpressible distinction, the declare-and-fail rule. A second copy of a rule is a second place for it to drift, and one of them had already drifted -- the @disabled-flags docstring spent fourteen lines disputing a claim ("a provider whose backend decides, such as one speaking OFREP, cannot") that the appendix no longer makes. So each docstring now says what the tag gates *here* -- which scenarios run, what a withholding skips, what Python's SDK makes of the question -- and links for the rest. What survives is what this repository owns: the two withholdings of @configuration-change that mean different things, the pytest-bdd feature-level marker mechanism that @standard-reasons depends on, the InMemoryProvider facts behind three of the self-tests' declarations, the measurement behind an empty INEXPRESSIBLE_CAPABILITIES, and the @reinitialization composition trap. Also drops one cross-language anecdote from extensions.py -- a same-named feature file on a second classpath root replacing the canonical one in another language. The rule it illustrates is stated right above it from this implementation's own uri derivation, and the anecdote is recorded where it happened. No behaviour change: 221 passed, 42 skipped, 2 xfailed, unmoved, and the module's code is identical line for line. Signed-off-by: Simon Schrottner --- .../contrib/tools/tck/capability.py | 437 ++++++------------ .../contrib/tools/tck/extensions.py | 4 +- 2 files changed, 133 insertions(+), 308 deletions(-) diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py index 0830465b6..e98dd5f56 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/capability.py @@ -19,13 +19,19 @@ class Capability(str, Enum): forcing such providers to fail scenarios they were never going to satisfy, each declares what it supports through :attr:`TckConfig.capabilities`. + `Appendix F + `_ + owns the vocabulary and the rules for declaring, including what a + declaration means, when to withhold one and when a + :class:`~.config.KnownDeviation` belongs beside it. The docstrings here say + what each tag gates *in this implementation* -- which scenarios run, what a + withholding skips, and what Python's SDK makes of the question -- and link + rather than restate. + Every capability corresponds to exactly one Gherkin tag. pytest-bdd turns those tags into pytest markers, and a scenario carrying a marker whose capability was not declared is skipped with the reason reported -- never - passed. A conformance suite that quietly goes green on scenarios it did not - run is worse than no suite at all. - - Scenarios with no capability tag are mandatory and always run. + passed. Scenarios with no capability tag are mandatory and always run. Two kinds of capability are refused rather than declared, and they are refused for different reasons and with different messages: @@ -38,18 +44,10 @@ class Capability(str, Enum): """Provider reaches its backend during initialisation, observably and promptly. Deliberately separate from :attr:`EVENTS`, because the two are independent in - both directions. - - An SDK dispatches ``PROVIDER_READY`` around ``initialize`` for *any* - provider, so a provider that declares ``EVENTS`` passes the readiness - scenario without demonstrating anything -- a ``NoOpProvider`` passes it - identically. Gating on ``EVENTS`` therefore made the scenario vacuous for - exactly the providers that declared it. - - Conversely a stateless provider -- one that resolves every flag with a fresh - request and holds nothing between them -- has a real initialisation to - verify while having no event stream of its own to declare ``EVENTS`` for. - Gating on ``EVENTS`` shut it out of a scenario it should be held to. + both directions: an SDK dispatches ``PROVIDER_READY`` around ``initialize`` + for *any* provider, so gating the readiness scenario on ``EVENTS`` made it + vacuous for exactly the providers that declared it, while a stateless + provider has a real initialisation to verify and no event stream of its own. Declare it if initialisation actually contacts the backend and its outcome, success or failure, is observable to the application. @@ -69,15 +67,13 @@ class Capability(str, Enum): repository's withholdings of it are real and neither is the other: * **A choice.** The OFREP adoption withholds it because every evaluation is - an independent HTTP request: there is no stream, no poll and no background - thread, so nothing is watching the backend and there is nothing to notice. - A provider built that way is not defective, and a + an independent HTTP request: nothing is watching the backend, so there is + nothing to notice. A provider built that way is not defective, and a :class:`~.config.KnownDeviation` there would assert a defect that does not - exist. The next evaluation does return the new value -- what is missing is - the *signal*, which is what the scenario asserts. + exist. * **A defect.** The in-memory self-test withholds it because the SDK's ``InMemoryProvider`` copies its flag mapping in the constructor and exposes - no way to change it, and `Appendix A + no way to change it, where `Appendix A `_ **requires** an SDK's in-memory provider to support updating the flag set and emitting this event. Tracked as `open-feature/python-sdk#620 @@ -95,27 +91,16 @@ class Capability(str, Enum): VARIANTS = "variants" """Provider names the variant it resolved. - Gated because a variant is optional rather than required. `Requirement 2.2.4 + Gated because a variant is optional rather than required: `Requirement 2.2.4 `_ - is a **SHOULD** -- in normal execution a provider "SHOULD populate the - resolution details structure's variant field" -- and ``types.md`` types the - field ``variant (string, optional)``. The same section adds that the value - "might only be meaningful in the context of the flag management system - associated with the provider". - - Some backends have no variant concept for a plain flag at all. Their - evaluation response carries no such key, so the provider never receives one - and no amount of seeding can produce one. Asserting a variant in every - evaluation scenario failed such a backend ten times over for something that - is not a defect and that no provider author can fix -- and left nothing to - record as a :class:`~.config.KnownDeviation`, because there was no - capability to hang one on. + is a **SHOULD** and ``types.md`` types the field ``variant (string, + optional)``, so a backend with no variant concept for a plain flag never + gives the provider one to report. Declaring it runs one Scenario Outline that asserts the variant for each of the eight flags whose variant name the canonical set fixes. Withholding it skips those rows with the reason and changes nothing else: the value - assertions live in untagged scenarios, because - `Requirement 2.2.3 + assertions live in untagged scenarios, because `Requirement 2.2.3 `_ makes the value a **MUST**. """ @@ -132,40 +117,6 @@ class Capability(str, Enum): nothing to act on, and no care in the provider produces a substitution it was never told to make. - Appendix F draws the line somewhere else, and what this suite measured does - not bear that out. The appendix has it that a provider whose backend decides, - "such as one speaking OFREP, cannot: the server never sees the caller's - default, so it has no way to return it". Both halves of that are observably - not the obstacle. flagd's RPC resolver is a remote evaluator by exactly that - description and satisfies the capability: the server answers reason - ``DISABLED`` with no variant and no value, and the resolver substitutes the - caller's default locally on the strength of that signal - (``resolvers/grpc.py``). flagd's OFREP endpoint answers the same flag with - ``{"reason": "DISABLED"}`` and no ``value`` and no ``variant`` -- the same - signal in another envelope -- and the Python OFREP provider already falls - back to the caller's default for the absent value. It fails these scenarios - for a reason unrelated to architecture, which its own suite records. - - So the tag is worth gating, but for the reason above rather than the one the - appendix gives, and that discrepancy belongs upstream rather than papered - over here. What it changes locally is only what a withheld declaration may be - read as: not necessarily an impossibility, so a reader has to look at the - adoption's own note for which it was. - - Withholding it **because the backend gives the provider nothing to act on** - needs no :class:`~.config.KnownDeviation`: a deviation records a gap in - behaviour the provider is *required* to have, and this one is optional, so an - entry would assert a defect that does not exist. - - A defect is the other case and does not get the same treatment, which this - docstring used to blur by sending both to "the adoption's own note". Where - the provider does attempt the resolution and gets it wrong -- the Python - OFREP provider above is exactly that, one unconditional index away from - passing -- Appendix F prefers the tag declared, the scenarios left to fail - and the deviation recorded beside them, because withdrawing the tag turns a - specific defect into a skip that reads as a design decision. Withholding is - for the provider that cannot attempt the behaviour at all. - Nothing in the specification says what a provider owes a disabled flag. `Requirement 1.4.7 `_ @@ -175,23 +126,30 @@ class Capability(str, Enum): Appendix F states the behaviour, the way it does for :attr:`NUMERIC_COERCION`, and gates it. + A remote evaluator is not shut out of this, which is worth knowing before + reading a withholding as an architectural impossibility: flagd's RPC resolver + satisfies the capability by substituting locally on the strength of the + server's ``DISABLED`` reason, and flagd's OFREP endpoint sends the same + signal in another envelope. The two adoptions in this repository record what + each one does with it. + Declaring it runs one Scenario Outline of four rows, over the four - ``disabled-*`` flags the canonical set added at spec revision ``009afe06``. - They mirror ``boolean-flag``, ``string-flag``, ``integer-flag`` and - ``float-flag`` exactly, differing only in ``state``, and each row's caller - default differs from the flag's configured value -- so a provider that - ignores the state returns the configured value and is caught on the value - alone, which rests on 2.2.3, a **MUST**. + ``disabled-*`` flags of the canonical set. They mirror ``boolean-flag``, + ``string-flag``, ``integer-flag`` and ``float-flag`` exactly, differing only + in ``state``, and each row's caller default differs from the flag's + configured value -- so a provider that ignores the state returns the + configured value and is caught on the value alone, which rests on 2.2.3, a + **MUST**. The rows assert the value and the absence of an error, and deliberately **not** the reason: pinning ``DISABLED`` here would rest on 2.2.5, a **SHOULD** that permits "some other string". It is pinned in ``reason.feature`` instead, which composes this tag with :attr:`STANDARD_REASONS` so that both must be declared before the reason is - asserted. No variant is asserted either, because a - disabled flag has resolved no variant and there is none to name -- so this - capability and :attr:`VARIANTS` do not compose, which is why the rows are not - part of the variant outline. + asserted. No variant is asserted either, because a disabled flag has resolved + no variant and there is none to name -- so this capability and + :attr:`VARIANTS` do not compose, which is why the rows are not part of the + variant outline. The SDK's own ``InMemoryProvider`` cannot declare this, and the reason is worth knowing before adopting it as a reference: ``InMemoryFlag`` accepts a @@ -206,137 +164,80 @@ class Capability(str, Enum): NUMERIC_COERCION = "numeric-coercion" """Provider coerces between integer and float only when lossless, else ``TYPE_MISMATCH``. - This is the one entry here that **the specification does not define**. - OpenFeature has a single numeric type on purpose -- ``number`` is "a numeric - value of unspecified type or size", and languages *may* differentiate between - integers and floats "as idioms dictate" -- so no requirement says what a - provider must do when a value does not fit the accessor it was asked through. - That gap is `open-feature/spec#430 - `_. - - The rule this capability is tested against is therefore **borrowed, not - normative**: lossless coercion is permitted, lossy coercion must fail. An - integral float such as ``10.0`` requested as an integer must succeed; ``0.5`` - must not. It comes from flagd's `numeric coercion ADR - `_, - which is scoped to flagd's own implementations, and the tag carries that name - -- it was ``@strict-numeric-typing`` -- because two vocabularies for one - observable property is worse than one borrowed name. - - **A provider that behaves differently is not violating the specification.** - So this is genuinely optional, rather than optional as a concession to a - defect -- but *which* of those a missing declaration means is not a free - choice, and this paragraph used to say it was. Appendix F's numeric-coercion - note settles it: a provider that **attempts** the coercion and gets one - direction wrong declares the capability, lets the lossy scenario fail, and - records a :class:`~.config.KnownDeviation` beside the failure -- because "it - coerces, and one direction is wrong" is exactly what a skip cannot say. - flagd is that provider (`open-feature/flagd#1996 - `_), and the flagd - adoption declares the tag and deviates rather than withholding. - - Withholding is for a provider that **cannot attempt** the behaviour: a - language with a single numeric type, where the distinction does not exist to - get wrong, or a provider that hands every variant back untouched and never - coerces at all. The SDK's own ``InMemoryProvider`` is the second kind, and - the paragraph below is what that looks like. - - Both halves have scenarios, and a provider declaring the tag must satisfy - all three. The lossy half asks for ``float-flag`` (``0.5``) as an integer - and expects ``TYPE_MISMATCH``; the lossless half asks for - ``integral-float-flag`` (``10.0``) as an integer and for ``integer-flag`` - (``10``) as a float, and expects both to succeed. Rejecting every float is - an easy way to pass the first, and the other two are what stop it. - - The SDK's own ``InMemoryProvider`` cannot declare this: it hands values + This is the one entry here that **the specification does not define**: the + rule is borrowed from flagd's `numeric coercion ADR + `_ + while `open-feature/spec#430 `_ + is open, so **a provider that behaves differently is not violating the + specification**. Appendix F's numeric-coercion note carries the rule, and + settles which of "declares and fails" and "withholds" a provider reaches for: + one that *attempts* the coercion and gets a direction wrong declares the + capability and records a :class:`~.config.KnownDeviation` beside the failing + scenario, and withholding is for one that cannot attempt the behaviour at + all. + + Both halves have scenarios, and a provider declaring the tag must satisfy all + three: the lossy half asks for ``float-flag`` (``0.5``) as an integer and + expects ``TYPE_MISMATCH``; the lossless half asks for ``integral-float-flag`` + (``10.0``) as an integer and for ``integer-flag`` (``10``) as a float, and + expects both to succeed. Rejecting every float is an easy way to pass the + first, and the other two are what stop it. + + The SDK's own ``InMemoryProvider`` is the withholding kind: it hands values back untouched and the client's type check is ``isinstance``-based, so ``10.0`` requested as an integer is a ``TYPE_MISMATCH`` rather than ``10``. - That is the provider declining to coerce, not the language refusing to ask: - a provider that does coerce returns an ``int`` and the same check passes it. - In a language with one numeric type the question could not be put at all, - which is why Appendix F names this as inexpressible there and why - :data:`INEXPRESSIBLE_CAPABILITIES` is empty here. The width of the integer - accessor is a separate property, and a separate capability: + That is the provider declining to coerce, not the language refusing to ask -- + a provider that does coerce returns an ``int`` and the same check passes it, + which is why :data:`INEXPRESSIBLE_CAPABILITIES` is empty here. The width of + the integer accessor is a separate property, and a separate capability: :attr:`LARGE_INTEGERS`. """ LARGE_INTEGERS = "large-integers" """Provider resolves integers up to 2^53 - 1 exactly. - A property of the language's SDK as much as of the provider, which is why - it is a capability rather than mandatory: Java's integer accessor is a - 32-bit ``Integer``, and a provider cannot resolve a value the accessor has - no room for. Every language can ask for 2^31 - 1, so that precision + A property of the language's SDK as much as of the provider, which is why it + is a capability rather than mandatory: a 32-bit integer accessor has no room + for the value. Every language can ask for 2^31 - 1, so that precision scenario is untagged; only the one asking for 2^53 - 1 carries this tag. Python's ``int`` is unbounded, so a Python provider declares it unless - something of its own -- a 32-bit field in its wire format, a float on the - way through -- narrows the value. Which makes this one of the two - capabilities Appendix F names as inexpressible somewhere and **not** here: + something of its own -- a 32-bit field in its wire format, a float on the way + through -- narrows the value, or unless the backend under test serves no such + flag for it to be asked about. Which makes this one of the two capabilities + Appendix F names as inexpressible somewhere and **not** here: :data:`INEXPRESSIBLE_CAPABILITIES` is empty in Python, and says on what - measurement. Nothing above 2^53 - 1 is asked for: - JavaScript cannot represent it, and what a provider owes a value that does - not fit the requested accessor is the open question in - `open-feature/spec#430 `_. + measurement. """ REINITIALIZATION = "reinitialization" """Provider can be initialised again after ``shutdown``, and serves flags afterwards. - Gated rather than mandatory because the specification permits reuse without - requiring it. `Requirement 2.5.2 + Gated rather than mandatory because `Requirement 2.5.2 `_ - says a provider **SHOULD** revert to its uninitialized state after - ``shutdown``, and its supporting text adds that "some providers **may** - allow reinitialization from this state". A provider that releases its client - on shutdown and declines to be started again is exercising a choice the - specification offers it, not exhibiting a defect -- so withholding this - capability needs no :class:`~.config.KnownDeviation` entry. - - The scenario was untagged until spec revision ``fc99d5ac``, on the reading - that reverting to the uninitialized state is observable as exactly one thing - -- being initialisable again. That inference does not hold, and asserting it - unconditionally reported a permitted choice as a conformance failure. A false - failure is the mirror image of a vacuous pass. - - Reverting the state is not separately observable either: a provider that - reverts but refuses reuse presents identically to one that did neither. So - the gated reuse scenario is the only assertion the requirement admits, and it - is worth keeping for the providers that do offer reuse -- releasing the client - on shutdown while leaving an initialised flag set behind is easy to write, - and leaves the provider evaluating against a closed connection rather than - failing outright. - - **This tag narrows :attr:`LIFECYCLE` rather than standing beside it.** The - scenario lives in ``lifecycle.feature``, which carries ``@lifecycle`` at the - feature level, so the scenario inherits that tag and carries both. The gate - skips a scenario if *any* capability gating it is undeclared, so reuse is - exercised only by an adoption declaring :attr:`LIFECYCLE` **and** this -- - declaring this one alone leaves the scenario skipped on ``@lifecycle``, and - the declaration unverified. Which is the trap worth naming: a provider that - withholds ``LIFECYCLE`` never ran this scenario, at this pin or the one - before it, so nothing about its behaviour on reuse has been observed either - way and there is no evidence on which to declare this. + permits reuse without requiring it: a provider that releases its client on + shutdown and declines to be started again is exercising a choice the + specification offers it, so withholding this capability needs no + :class:`~.config.KnownDeviation` entry. + + **This tag narrows :attr:`LIFECYCLE` rather than standing beside it**, and + that is the trap worth naming. The scenario lives in ``lifecycle.feature``, + which carries ``@lifecycle`` at the feature level, and the gate skips a + scenario if *any* capability gating it is undeclared -- so declaring this one + alone leaves the scenario skipped and the declaration unverified. A provider + that withholds ``LIFECYCLE`` has therefore never run this scenario, and has + no evidence on which to declare this one either way. """ TARGETING = "targeting" """Provider resolves a flag differently for a matching evaluation context. - Reserved and undeclarable until spec revision ``26362f85``, on the reading - that targeting is backend evaluation logic and therefore out of scope. The - scope argument still holds -- what the three scenarios test is not how a - backend evaluates a rule -- but the conclusion did not: they exist to show - that the **context reached the backend at all**, which is a property of the - provider and of nothing else. - - ``targeting-key-flag`` is the one flag in the canonical set with a rule, and - it is what makes passthrough observable without an echo endpoint on the - control API: a matching context resolves ``hit`` where anything else - resolves ``miss``, so a provider that drops the context on the floor is - caught by the resolved value itself. The rule is specified by behaviour - rather than by syntax -- resolve ``hit`` when the targeting key is exactly - ``5c3d8535-f81a-4478-a6d3-afaa4d51199e`` -- so a backend expresses it - however it expresses targeting. + What the three scenarios test is not how a backend evaluates a rule: they + exist to show that the **context reached the backend at all**, which is a + property of the provider and of nothing else. ``targeting-key-flag`` is the + one flag in the canonical set with a rule, and it is what makes passthrough + observable without an echo endpoint on the control API -- a matching context + resolves ``hit`` where anything else resolves ``miss``. The three scenarios are the matching context, the non-matching one and no context at all. The second and third are not padding: a provider that always @@ -344,12 +245,9 @@ class Capability(str, Enum): evaluate a rule with no targeting key present is caught by the third. Two more scenarios carry this tag alongside :attr:`STANDARD_REASONS`, in - ``reason.feature``, asserting ``TARGETING_MATCH`` for the hit and - ``DEFAULT`` for the miss. They need both: a provider with no targeting has - no rule to match, so there is no ``TARGETING_MATCH`` for it to report and - the scenario would fail it for an absence rather than a defect. Declaring - this capability alone leaves them skipped, and changes nothing about the - three above. + ``reason.feature``, asserting ``TARGETING_MATCH`` for the hit and ``DEFAULT`` + for the miss; declaring this capability alone leaves them skipped and changes + nothing about the three above. Declare it if the backend under test can express that rule and the provider forwards the targeting key. A backend with no targeting at all leaves it @@ -363,69 +261,17 @@ class Capability(str, Enum): **A claim, not an exemption.** `Requirement 2.2.5 `_ - is a **SHOULD**, and it goes further than 2.2.4 does: it lets a provider - populate ``reason`` with one of the listed values *"or some other string - indicating the semantic reason for the returned flag value"*. A provider - whose backend reports vendor-specific reasons is therefore conformant, and - asserting an exact reason against it would fail it for something the - specification permits. - - An earlier revision of the suite did exactly that, in thirteen places across - ``evaluation.feature``, ``errors.feature`` and ``lifecycle.feature``, and - Appendix F recorded the narrowing as a deliberate exception. It is not one - any more. It bought very little -- every canonical flag resolves to a value - distinct from the caller's default, so a provider that silently falls back - is already caught by the value assertion, and the reason only said *why* it - failed -- and of the thirteen, five sat beside an error-code assertion that - already carries the **MUST**, while the other eight asserted ``STATIC``, the - one reason the specification genuinely leaves open. - - So the reasons now live in ``reason.feature``, gated as a whole at the - feature level. Declaring this capability is a provider saying "I use the - standard vocabulary with the standard meanings", and that file is what - checks the claim. A provider that does not declare it **loses nothing**: its - values, variants and error codes are asserted everywhere else, on **MUST** - requirements. What the declaration adds is something a report's reader can - act on -- anyone building telemetry, dashboards or debugging on ``reason`` - can see that the vocabulary was verified rather than assumed. - - The meanings are the content of the claim, and they constrain nobody who - does not make it: - - * ``STATIC`` -- the flag was resolved from configuration and carries no - targeting rule; - * ``TARGETING_MATCH`` -- a targeting rule matched the evaluation context; - * ``DEFAULT`` -- a targeting rule exists and did not match; - * ``DISABLED`` -- the flag is disabled in the management system; - * ``ERROR`` -- the evaluation failed, and an error code is reported with it. - - ``STATIC`` for the first row is the call worth flagging. ``types.md`` types - ``DEFAULT`` as *"no dynamic evaluation occurred **or** dynamic evaluation - yielded no result"*, which a rule-less flag satisfies as readily as - ``STATIC`` does -- two providers can disagree here and both conform. A - provider that answers ``DEFAULT`` for a rule-less flag is not defective; it - does not use the standard meanings, and should not declare the tag. - - ``ERROR`` is the row where the suite's subject is blurred, and it is - asserted anyway. The other four rest on `Requirement 1.4.7 - `_, - which makes the SDK propagate the provider's reason -- but only *"in cases of - normal execution"*. Abnormal execution is 1.4.9, a **SHOULD** on the *SDK* to - "indicate an error", and nothing requires the provider's reason to survive. - So a passing ``ERROR`` scenario establishes that what reached the - application is coherent, not that the provider produced it. It is still - worth asserting, because the pair is what carries the meaning: the error - code alone is already covered for every provider by ``errors.feature``, - ungated and on a **MUST**, and the reason alone could have been written by - the SDK. An evaluation reporting ``FLAG_NOT_FOUND`` with reason ``STATIC`` - is incoherent whoever wrote it. - - ``SPLIT``, ``UNKNOWN``, ``CACHED`` and ``STALE`` are not asserted. The first - two have no scenario that produces them; ``CACHED`` needs a repeat - evaluation, which nothing here performs without a configuration change in - between, and belongs behind the reserved :attr:`CACHING`; ``STALE`` needs a - scenario asserting what a provider serves *during* an outage, which is the - same gap. + is a **SHOULD** that lets a provider populate ``reason`` with *"some other + string indicating the semantic reason for the returned flag value"*, so a + provider whose backend reports vendor-specific reasons is conformant and is + not expected to declare this. Declaring it says "I use the standard + vocabulary with the standard meanings", and ``reason.feature`` is what checks + the claim; Appendix F's ``@standard-reasons`` section is where the meanings + are fixed, including the two rows -- ``STATIC`` for a rule-less flag, and + ``ERROR`` -- that are calls rather than consequences. + + A provider that does not declare it **loses nothing**: its values, variants + and error codes are asserted everywhere else, on **MUST** requirements. **Tags compose, and here that is load-bearing.** ``TARGETING_MATCH`` cannot be observed without targeting and ``DISABLED`` cannot be observed unless the @@ -481,20 +327,14 @@ def __str__(self) -> str: """Capabilities that exist in the vocabulary and gate no scenario. They are documented so the vocabulary has a place for them when scenarios exist, -and until then they **must not be declared** and must not appear in a conformance -report's declaration. Nothing carries the tag, so declaring it cannot be -verified, cannot produce a skip, and tells a reader of the report only that -something was claimed and nothing examined. +and until then they **must not be declared** -- nothing carries the tag, so +declaring it cannot be verified and cannot produce a skip. Listed once, here, and read everywhere else -- by :data:`DECLARABLE_CAPABILITIES`, by :attr:`Capability.reserved` and by the validation in :class:`~.config.TckConfig` -- so that the set and the rule cannot -drift apart. - -``@caching`` is the only one left. :attr:`Capability.TARGETING` was here until -spec revision ``26362f85`` gave it scenarios, and leaving a tag reserved once it -has them would be the mirror of the mistake this set exists to prevent: a -capability that *can* be verified and is refused the chance. +drift apart. A reservation expires in the specification repository rather than +here, which is what :func:`expired_reservations` exists to notice. """ INEXPRESSIBLE_CAPABILITIES: Mapping[Capability, str] = MappingProxyType({}) @@ -518,29 +358,20 @@ def __str__(self) -> str: missing one is discovered by an adopter publishing a claim no scenario could have examined. -**Not the same thing as a reservation, and the difference is what the two -messages have to carry.** A reserved capability is global and temporary -- no -scenario anywhere carries the tag, and the reservation expires the moment the -specification writes one. An inexpressible capability is one language's and -permanent: the scenarios exist, other languages run them and pass them, and -nothing changes until the SDK does. A reader seeing a capability missing from a -report has to be able to tell *"this provider declined"* from *"no provider in -this language can be asked"*, because only the first says anything about the -provider. Hence a mapping rather than a set: the value is the property of the -SDK that puts the question out of reach, and it is the half of the message an +**Not the same thing as a reservation**, and the value is what carries the +difference: a reader seeing a capability missing from a report has to be able to +tell *"this provider declined"* from *"no provider in this language can be +asked"*. Hence a mapping rather than a set -- the value is the property of the +SDK that puts the question out of reach, which is the half of the message an adopter could not have worked out for themselves. A capability belongs here only when **no** provider in this language could ever -satisfy it. A provider that gets the answer wrong is a different thing entirely -and belongs nowhere near this mapping. flagd's two Python resolvers answer the -three ``@numeric-coercion`` scenarios differently from each other: in-process -refuses ``0.5`` as an integer and widens ``10`` to a float, while RPC widens -``10`` and silently narrows ``0.5`` to ``0``. That split is a defect in one -resolver of one implementation, and **both** resolvers declare the tag: the one -that narrows carries the :class:`~.config.KnownDeviation` and the one that does -not carries none, which is the shape Appendix F prefers. Listing it here would -say the question cannot be asked -- and two resolvers of one provider giving -different answers to it is the proof that it can. +satisfy it. A provider that gets the answer wrong is a different thing entirely: +flagd's two Python resolvers answer the ``@numeric-coercion`` scenarios +differently from each other, and **both** declare the tag, the one that narrows +carrying the :class:`~.config.KnownDeviation`. Listing it here would say the +question cannot be asked -- and two resolvers of one provider giving different +answers to it is the proof that it can. Never overlaps :data:`RESERVED_CAPABILITIES`: a tag no scenario carries is reserved, whatever any SDK could express about it. @@ -558,12 +389,10 @@ def __str__(self) -> str: set surfaces gaps; widening towards it hides them. It excludes the reserved capabilities rather than spanning the whole enum, and it -is named for what it is rather than for "all", because the declare-everything -convenience is exactly how a reserved tag reaches a report by accident: an +is named for what it is rather than for "all", because a declare-everything +convenience is exactly how a reserved tag reaches a report by accident -- an adopter writing "every capability except X" picks up every reserved tag on the -way past, which is how one implementation came to report ``@targeting`` and -``@caching`` as declared without anyone deciding to claim them -- back when both -were reserved. +way past. It excludes :data:`INEXPRESSIBLE_CAPABILITIES` for the same reason and one more: that set is empty in Python, so a default spanning the whole enum would look @@ -599,20 +428,18 @@ def capability_for_tag(tag: str) -> Capability | None: def expired_reservations(tags: typing.Iterable[str]) -> tuple[Capability, ...]: """Reserved capabilities that the tags handed in turn out to carry. - A non-empty answer means some scenario is both unrunnable and unclaimable. - Declaring a reserved capability is refused, so the capability gate skips - every scenario carrying its tag, and the report says a gap exists where the - provider may well have none. Appendix F names that the unclaimable - capability, and it is the quieter mirror of declaring a capability nothing - verifies: nobody can claim the tag, so nothing else about the run changes. + A non-empty answer means some scenario is both unrunnable and unclaimable: + declaring a reserved capability is refused, so the gate skips every scenario + carrying its tag, and the report says a gap exists where the provider may + well have none. Two things put a reserved tag on a scenario and this reports only that one of them happened. Either :data:`RESERVED_CAPABILITIES` is out of date -- - the scenarios the tag was held open for now exist, so the capability can be - verified and an adoption should be allowed, and required, to say whether it - has it -- or an adopter has used a reserved name for a tag of their own. - Telling the two apart is the caller's, because the caller is what knows - where the tags came from: the remedies differ and the consequence does not. + the scenarios the tag was held open for now exist, so an adoption should be + allowed, and required, to say whether it has the capability -- or an adopter + has used a reserved name for a tag of their own. Telling the two apart is the + caller's, because the caller is what knows where the tags came from: the + remedies differ and the consequence does not. Compared against tags rather than against a second list, because a reservation expires in the specification repository while this set lives diff --git a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py index 473264570..6424ee55c 100644 --- a/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py +++ b/tools/openfeature-tck/src/openfeature/contrib/tools/tck/extensions.py @@ -35,9 +35,7 @@ arrives as ``gherkin/errors.feature`` -- the same uri as a canonical file. A record of what ran holds one copy of a feature file per uri, so the second file is never read and its scenarios are attributed to the first one's or to nothing -at all. Java hit the same thing by a different route: a same-named feature file -in a second classpath root replaced the canonical one outright and the suite went -green having run the adopter's version. +at all. The derivation is public, and the two problems it cannot rule out are reported rather than raised, because the consumer of all of this is a conformance report