test(flagsmith): run the provider conformance suite against the testbed

[aepfli]

Runs the OpenFeature Provider Conformance Suite against the Flagsmith Java provider.

Stacked on feat/provider-tck. Companion to go-sdk-contrib#959 and js-sdk-contrib#1623 — same container, same scenarios.

Result: 31 pass, 5 fail, 29 skipped of 65

	Go	Python	Java	JS
passed	43	28	31	29
failed	3	5	5	4
skipped	19	23	29	32
of	65	56*	65	65

* Python's TCK has not picked up @standard-reasons yet.

@standard-reasons is what changed this run. It was 24 pass / 12 fail before that capability existed; eight of those twelve failures were this provider not populating the resolution reason, and they are now skips carrying a recorded reason rather than bare failures.

@standard-reasons — withheld, with a deviation

This provider never populates the resolution reason. Every evaluation returns null, so not one of the reasons the capability claims is reported. The resolved values are correct throughout; only the reason is missing.

2.2.5 makes the reason a SHOULD, which is exactly why this is a withheld claim rather than a failure. What makes it a defect worth recording is that the Go Flagsmith provider reports STATIC, DISABLED and TARGETING_MATCH against the identical backend — so this is a gap rather than a considered choice.

The 5 remaining failures

• Two are unreadable floats. Flagsmith has no float type — feature_state_value is natively boolean, integer or string — so every float is stored as a string. Go's provider parses it back ("Because We store floats as string"); this one type-checks and falls through to the caller default, so getFloatValue never works against Flagsmith. Python has the same gap; the @numeric-coercion deviation now records this as measured rather than predicted.
• Two are the shared type-system pair. float-flag and object-flag requested as a String succeed rather than reporting TYPE_MISMATCH, because on this backend both genuinely are strings. Not a provider bug in any of the four languages.
• One is object-flag not resolving as a structure.

@large-integers — refused by the TCK, and correctly

The base now refuses this capability in Java rather than leaving it to the adopter: Java's integer accessor is a 32-bit Integer, so 2^53−1 cannot be asked for at all. That is a property of the SDK, not of the provider, and it is the same reason Java withholds it for flagd.

Worth knowing alongside it: Value.asInteger() silently truncates a Long to its low 32 bits — 4294967301 becomes 5, 2^53−1 becomes −1 — with no error. That is a defect in the Java SDK rather than anything this PR touches, but it is why the refusal matters: declaring the capability would not have errored, it would have returned a plausible wrong number.

Other capabilities

Declared: @object, @targeting, @disabled-flags.

@disabled-flags holds because Flagsmith models a feature state as enabled plus a value, so the canonical set's four disabled-* flags map straight onto it, and this provider returns the caller default with reason DISABLED. Go agrees; Python and JS both raise GENERAL and withhold it.

@variants is withheld: Flagsmith has no variant concept for a plain feature, and the evaluation response carries no variant key. Permitted rather than defective — 2.2.4 makes it a SHOULD — so no deviation.

The lifecycle and event capabilities are withheld because this provider has no observable initialisation for the suite to assert against.

A finding about the TCK, not the provider

tools/tck exports jackson-databind 2.22.1, while flagsmith-java-client 7.4.3 pins jackson-annotations and jackson-core at 2.15.2. Maven's nearest-wins assembles databind 2.22 against annotations 2.15, and the Flagsmith client's ObjectMapper dies with ClassNotFoundException: com.fasterxml.jackson.annotation.JsonSerializeAs.

Verified this is the TCK's doing: with the TCK dependency removed the provider's own 35 tests pass; with it added, 22 error. Re-tested after the TCK moved testcontainers to provided/optional — still present, so it wants its own fix upstream: shade Jackson, or stop exporting a hard databind version.

The pin here is the workaround. It is compile scope and has to be, because the provider's own main source imports com.fasterxml.jackson.databind. It cannot be aligned upward — jackson-annotations has no 2.22.1 release, it tracks its own version line — so that route needs the Jackson BOM.

The backend

aepfli/flagsmith-tck-testbed 0.1.0, pinned rather than :latest because four adoptions pull it and a mutable tag lets a push change four PRs' results with no diff to explain it.

Why this is a draft

• The testbed image lives in a personal namespace.
• The Jackson conflict wants a decision upstream rather than a per-provider pin.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}