feat(provider-tck): add Java conformance suite for OpenFeature providers

[aepfli]

Part of open-feature/spec#417 (cross-language tracking) via #1829 (Java implementation issue). The language-agnostic artifacts live in spec#423; the report envelope in spec#425.

Closes #1829

Draft — opened for review of the approach. The module passes the full codequality,deploy gate. The open questions are where I would most value pushback.

The PR chain

The same four-branch shape in every language, so understanding one is understanding all four:

main
 └── #1830  feat/provider-tck           this PR — the suite, both control paths, the spec
      │                                  submodule, the extension point, and the entire
      │                                  adopter-facing API
      ├── #1841  feat/provider-tck-report   Cucumber Messages and the report envelope
      └── #1847  feat/provider-tck-flagd    flagd adoption, both resolvers
           └── #1840  feat/provider-tck-ofrep   OFREP adoption

report and flagd are siblings, not a sequence. The follow-ups fill gaps without changing the
adopter-facing API, so neither blocks the other and either can land first. The suite works without
reporting, which is why reporting is not in this PR.

The seam: this branch owns everything an adopter declares — capabilities(),
knownDeviations(), configuration(), BackendControl.controlApi(), the reserved-capability rules
and the tag lookup. Reporting is purely additive over that, so adopting the suite and emitting a
report ask for the same declaration and nothing more.

What this is

A conformance suite any OpenFeature provider can adopt to check that it implements the provider
contract in the specification. OpenFeature's central promise is that swapping providers does not
change application behaviour; nothing verifies that today, and every provider tests differently.

The new tools/tck module carries the Cucumber step definitions and an abstract JUnit Platform
Suite that owns the whole test lifecycle. It does not carry the Gherkin, the canonical flag set
or the control API description: those are language-agnostic and live in open-feature/spec, reaching
this module through a submodule and packaged into the JAR at build time. A copy would be a second
place for conformance to drift, which is the one thing this suite exists to prevent. Adopters need
no submodule.

flagd is the first adopter, in both resolver modes. flagd-testbed is not modified and the existing
flagd e2e suites are untouched.

The adoption surface

Four methods with no default; everything else is a convention you can override. The whole of flagd's
RPC mode, on top of a shared abstract class:

public class RpcTest extends AbstractResolverTest {

    @Override protected Config.Resolver resolver() { return Config.Resolver.RPC; }

    @Override protected int backendPort() { return 8013; }
}

The shared part supplies the four required methods:

@Override public File composeFile() { ... }                      // the stack to run
@Override public List<Integer> backendPorts() { ... }             // ports the provider connects to
@Override public FeatureProvider createProvider(BackendEndpoint endpoint) { ... }
@Override public FeatureProvider createUnavailableProvider() { ... }

createProvider is a factory taking an endpoint rather than a field, because container host ports
are assigned dynamically and do not exist until the stack is up. Conventions with working defaults:
backendService() (backend), controlPort() (8080), additionalPorts(),
backendConfiguration() (default), startupTimeout() (60s).

The suite owns the container stack. Starting Compose, discovering mapped ports, building the
control client, waiting for the control API, tearing down — all of it. An adopter writes a Compose
file and the methods above. This is now required by Appendix F, after three of the four languages
originally shipped only the control-API client and left orchestration to the adopter, at which point
every adoption hand-rolled between 130 and 460 lines of the same wrapper.

A provider with no backend supplies its own BackendControl instead and never touches Compose.

Declaring what a provider can and cannot do

Two separate statements, because conflating them is how a conformance suite starts lying.

capabilities() names the optional parts of the contract this provider supports, from fourteen tags
(@lifecycle, @events, @stale, @targeting, @disabled-flags, @numeric-coercion,
@standard-reasons, …; @caching is reserved and cannot be declared). A scenario gated on an
undeclared capability is reported
skipped with its reason, never passed.

@large-integers cannot be declared in Java at all, and the suite refuses it rather than
expecting adopters to know that. The integer accessor is a 32-bit Integer, so 2^53 − 1 cannot be
asked for by any Java provider — it is a property of the SDK and says nothing about the provider.
Appendix F makes this the implementation's call rather than each adopter's, and naming it now fails
at configuration time with a message saying why. Its scenario stays gated rather than becoming
ungated, so it is skipped with a reason distinct from "the provider declined" — a reader of a Java
report can tell the two apart. Five places in this repository previously stated that fact, including
a README section instructing adopters to withhold it.

knownDeviations() says the provider fails something it is required to do. Appendix F settles which
of its two shapes to prefer: declare the capability, let the scenario fail, and record the deviation
beside the failure — rather than withdrawing the capability so the scenario skips. Withdrawing to
turn a failure into a skip hides a defect behind something that looks deliberate.

flagd is the worked example. It narrows 0.5 to 0 when a float flag is read through the integer
accessor, with no error code. Earlier revisions of this PR withheld @numeric-coercion; it now
declares it and carries a tracked deviation, because the widening scenario passes — so flagd
does coerce and gets one direction wrong, and a skip could not distinguish that from declining to
coerce at all.

reason is asserted in two places only, and they mean different things. Requirement 2.2.5 is a
SHOULD permitting "some other string", so the scenarios that resolve a value no longer assert a
reason at all — an exact-match assertion there fails providers the specification permits. What
replaced it is a capability. @standard-reasons declares that this provider speaks the
specification's own vocabulary, and a feature file then checks the whole of it: STATIC,
TARGETING_MATCH, DEFAULT, DISABLED. Undeclared, those scenarios skip with their reason, and
nothing else in the suite cares what a provider calls a reason.

The error cases are not part of that bargain and are asserted unconditionally. Requirement 1.4.8
makes an error code mandatory in abnormal execution, so ERROR there is not a dialect — a reason
that disagrees with the error code shipped beside it is a contradiction. The assertion is on the two
agreeing, not on which of them is authoritative.

The two invariants a TCK implementation gets wrong

Both are normative in the control API description.

Containers are never stopped or restarted mid-suite. Unavailability is simulated inside the
running stack. This is portability, not preference: container runtimes assign host ports dynamically
and do not reliably preserve them across a restart, and which bindings preserve them differs by
language — so a TCK that restarts containers works in one language and mysteriously fails in
another.

A control endpoint that changes flag state must not return until that state is being served.
Returning on accepted rather than applied pushes a race onto every caller, and the caller cannot
close it: a suite cannot distinguish "the backend has not caught up" from "the provider resolved the
wrong value". Earlier revisions of this PR slept 50ms after every control call; that is gone. The
wait is the endpoint's obligation, and where a backend breaks it the wait belongs in that adoption,
named and citing the defect — see AbstractResolverTest, which does exactly that for
flagd-testbed#394.

Running it

A conformance adoption is a directory, not a naming convention. It lives in the module's own
…/tck/ package beside the e2e suite rather than inside it, and every selector is now a path:
**/tck/*.java for the tck profile's includes, **/e2e/*.java,**/tck/*.java for the default
build's exclusions. providers/ofrep's e2e package is gone entirely — the conformance suite had
been its only member, which is the argument in its clearest form. The class names lost the Tck the
selector used to be spelled with: RpcTest, InProcessTest, OfrepTest. *Test stays, because
Surefire's default includes need it and that is a different thing from the selector being removed.

The adoption suites are excluded from the default build via testExclusions in the adopter's
POM, and a maintainer runs them through the dedicated tck profile:

mvn -pl tools/tck -am -DskipTests install     # once — tools/tck is not published yet
mvn -Ptck -pl providers/flagd test

Do not add -am to the second command. It drags tools/tck and tools/flagd-core into the
reactor and runs their unit suites first, which puts a conformance failure and a unit-test failure
back on one signal — the thing a separate step exists to prevent.

An earlier revision of this section documented mvn -P e2e test, and that was wrong in the worst
direction. Measured against a real run, it executes the three legacy e2e suites — 788 tests — and
TckTest appears nowhere in the 7309-line log. -P e2e is in fact the one flag that guarantees
the conformance suites do not run. The tallies below were always produced by a different invocation
and the adoption README documented that one correctly; only this description was wrong.

Appendix F carries the reasoning for excluding it: an adoption suite's honest output is red while
real gaps remain — filed provider defects, missing backend fixtures — so making it a required gate
forces someone to silence it, and the cheapest way to silence a conformance suite is to stop asking
the question.

So read the tally below rather than the green check. CI green on the adoption PRs means the
suite did not run. It does still compile in the default build, which is deliberate and which
testExclusions preserves by being a Surefire and not a compiler exclusion: a suite that has
quietly stopped building against its own harness is a worse failure than one that runs and fails.

The gating half of this is provisional and under discussion. The suite is excluded because its
honest output is red — but that is only true while a run is judged by "zero failures". Judged
instead by whether its results match its declaration — every failure covered by a declared
deviation, every skip gated by an undeclared capability, every deviation still failing something —
a healthy adoption is green in its steady state, deviations included, and this could be a required
gate. Proposed in
open-feature/spec#417;
nothing here depends on the outcome. What is settled either way is that the conformance suite gets
a step of its own rather than being folded into the provider's e2e suite.

Verification

At fe4ac3be, run locally:

Check	Result
mvn --projects tools/tck -P codequality,deploy clean verify	passes — checkstyle, PMD, SpotBugs, Spotless/Palantir, javadoc failOnWarnings=true
Module's own suite	246 tests green, no Docker
RpcTest	65 scenarios — 59 passed, 4 failed, 2 skipped
InProcessTest	65 scenarios — 59 passed, 4 failed, 2 skipped
Existing flagd e2e suites	unaffected and green

The four failures are one real provider defect (the float-to-int narrowing above, declared as a
deviation) and three backend fixture gaps that
flagd-testbed#392 fills. The two skips are
the scenarios gated on @reinitialization and @large-integers, the two capabilities flagd
withholds — each reported with its reason rather than passed.

Six lifecycle scenarios also run without Docker, through a test-scoped controllable provider, so
shutdown and re-initialisation are exercised in a normal build rather than only through the
containerised adoption.

Open questions

1. Should testcontainers be a compile dependency of tools/tck? It is provided/optional
   here. A separate tck-testcontainers artifact would drag createProvider(BackendEndpoint) out
   of the core, so I did not split it — but four languages answered this four ways and none is
   obviously wrong.
2. Nothing validates an emitted report against the schema in CI, in any language. The report
   branch checks fields by hand. The schema is what makes reports comparable across languages, so
   this is the weakest link in the design and it wants one answer rather than four.
3. Scenario scope. A representative subset covering each architectural mechanism once. Is the
   boundary in the right place?

Known gaps

Listed in full in tools/tck/README.md, and the same five in every language: whole-context
passthrough beyond the targeting key, targeting and bucketing correctness (out of scope by design),
caching, per-flag set/remove control operations, and hooks.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}