[Tracking] A cross-language conformance suite for OpenFeature providers

[aepfli]

The problem

OpenFeature's central promise is vendor neutrality: swapping providers must not change application
behaviour. Nothing currently verifies that.

Every provider tests differently. Some have thorough suites, some have almost none, and no two
agree on what "implements the provider contract" means. The consequences are real and already
visible:

• Providers disagree on whether a type mismatch returns the code default or throws.
• Providers disagree on whether an unreachable backend yields ERROR promptly, hangs, or throws
  out of provider registration.
• Providers disagree on which lifecycle events fire, and in what order, around a disconnect.
• Some providers silently coerce between numeric types instead of reporting TYPE_MISMATCH.

Every one of these is invisible until an application swaps providers in production. The spec
describes the correct behaviour in prose; there is no executable definition of it.

The proposal

A language-agnostic conformance suite for providers — a set of Gherkin feature files, a
standardised backend control API, and a canonical flag set — that each language's TCK implementation
packages and runs against real providers.

We have built a working proof of concept in Java, using flagd as the first provider under test. It
finds real bugs already (see Findings). This issue is the cross-language
tracking issue; it is opened for discussion before anything is merged into the spec repo.

What lives in this repository

Three artifacts, which must travel together:

Artifact	What it is
gherkin/*.feature	The canonical test cases — the executable definition of the provider contract
openapi/control-api.yaml	What a backend under test must expose so a TCK can drive it
flags/canonical-flags.json	The flag set those test cases assume

They belong here rather than in any single language's repository because:

1. They are the contract. If every language's TCK writes its own scenarios, we get N different
   definitions of conformance and are no better off than today.
2. Backend vendors implement the control API in their own language. flagd's testbed is Go. It
   should not have to read a Java repository to learn what POST /restart must do.
3. They cannot be split. A feature file that evaluates boolean-flag is meaningless without the
   flag definition; a disconnect scenario is meaningless without the endpoint that produces the
   disconnect. Splitting them across repositories guarantees drift.

Each language's TCK then vendors this repo (git submodule or equivalent) and packages the files into
its own distribution format, so adopting a TCK never requires a consumer to check out a submodule of
their own.

What lives in each language repository

The runner: step definitions, an abstract base test class owning the container lifecycle and control
API client, and whatever adoption ergonomics that language's testing culture expects.

Scope: the provider contract only

In scope — how a provider maps backend responses onto the spec's types and lifecycle:

• typed resolution details: value, variant, reason, error code
• keeping integer and float distinct, with no silent coercion
• error handling: type mismatch and unknown flag return the code default, report the right error
  code, and never throw
• lifecycle: reaching READY; settling into ERROR against an unreachable backend
• events: PROVIDER_READY, PROVIDER_ERROR, PROVIDER_STALE, PROVIDER_CONFIGURATION_CHANGED
• that a signalled configuration change is actually applied on re-evaluation

Out of scope — deliberately:

• backend evaluation logic, targeting and bucketing correctness. That is the backend's contract, not
  the provider's. Every flag in the canonical set resolves to its default variant.
• the provider↔backend wire protocol.
• SDK behaviour, which belongs to each SDK's own suite.

Two design decisions worth discussing

1. The backend under test is a Compose stack, not an image

The TCK does not start a container; it starts a docker-compose.yaml the vendor supplies. This lets
vendors compose a proxy, an edge service, or several containers — a single image cannot express
"backend behind a toxiproxy". The TCK only relies on conventions: a known service name, a known
container-internal port for the control API, and dynamically mapped external ports.

2. Containers are never stopped or restarted mid-suite

This is the most important normative requirement and the least obvious one:

Backend unavailability MUST be simulated inside the running stack — a process kill, a proxy toxic,
a socket block. Implementations MUST NOT stop, kill, remove or recreate a container while the
suite is running.

The reason is portability, not preference. Testcontainers cannot reliably preserve dynamically
mapped host ports across a container restart, and which bindings preserve them differs by
language. A TCK that restarts containers works in one language and mysteriously fails in another —
the worst possible property for a cross-language conformance suite.

flagd's testbed already does the right thing: it kills and restarts the flagd process inside a
container that keeps running. That is the reference behaviour.

The control API

Derived from the endpoints flagd's testbed "launchpad" already implements, so flagd conforms today
without modification.

Endpoint	Status	Semantics
POST /start?config={name}	required	start the backend, seed flags to that config's baseline
POST /stop	required	make the backend unreachable; must not stop the container
POST /restart?seconds={n}	required	bounded outage; flag state preserved across it
POST /change	required	change changing-flag's resolved value without a restart
POST /reset	optional	restore baseline with no outage; falls back to /start
GET /healthz	optional	readiness; falls back to a TCP port check

A second normative rule, which the PoC depends on for scenario isolation:

/start resets flag state to the named configuration's baseline. /restart preserves it. An
outage must be observable as a change in availability, never as a change in flag values.

The two optional endpoints exist so that flagd — which implements neither — can adopt the suite
unchanged, while giving vendors building a new testbed something better to aim at.

Capability tags

Not every provider implements every optional part of the spec. Scenarios exercising an optional
capability carry a tag; a provider declares what it supports and the rest are reported as
skipped, with the reason printed — never silently passed. A conformance suite that quietly goes
green on scenarios it did not run is worse than no suite.

Current tags: @events, @stale, @configuration-change, @object, @unavailable,
@strict-numeric-typing, plus reserved @targeting and @caching.

Findings from the PoC

Running the PoC against the flagd Java provider, first attempt, already surfaced a real defect:

flagd silently narrows a float flag to an integer. Evaluating float-flag (0.5) through the
integer API returns 0 with no error code at all, rather than TYPE_MISMATCH with the code
default. The application sees a plausible value and no indication anything went wrong — the worst
failure mode a feature flag can have. This is precisely the class of bug that only a shared,
adversarial suite finds, since the provider's own tests encode the provider's own assumptions.

That single finding is, we think, the argument for this issue.

Open questions for discussion

1. Directory layout in this repo. provider-tck/{gherkin,flags,openapi}/? Something else?
2. Versioning. Should the suite be versioned independently of the spec, so a provider can claim
   "conformant to provider-tck 1.2" without tracking spec revisions? Related: how do we add
   scenarios without instantly breaking every adopter's build?
3. The canonical flag set's format. The PoC expresses it in flagd's flag-definition format,
   because that is the only widely implemented vendor-neutral format today. Is that acceptable as a
   description that vendors translate, or do we want a neutral schema?
4. Is @strict-numeric-typing a capability or a hard requirement? It is modelled as a capability
   so providers with the defect can adopt today and see it reported as an explicit skip. That is
   pragmatic, but it does put a spec violation on the same footing as an optional feature. Better
   idea welcome — perhaps a separate "known deviations" concept.
5. Context passthrough. The suite cannot currently assert an evaluation context reached the
   backend. That needs an echo operation (GET /last-evaluation?) on the control API. Worth adding
   before the first stable release?
6. Reason values. The PoC asserts STATIC for flags with no targeting rules. Is that
   universally correct across providers, or does it need to be a per-provider expectation?
7. Who owns conformance claims? Is there a badge, a report format, a listing on openfeature.dev?
   Or is passing simply its own reward for now?

Per-language tracking

• Java — proof of concept complete (see the linked implementation issue)
• JavaScript / TypeScript
• Go
• .NET
• Python
• PHP
• Ruby
• Kotlin / Swift (client-side)

Prior art in this ecosystem

• open-feature/test-harness — the flagd-oriented
  Gherkin suite and testbed this work generalises.
• open-feature/flagd-testbed — the launchpad
  control API the standardised one is derived from.
• dev.openfeature.contrib.tools:flagd-api-testkit in java-sdk-contrib — an existing "abstract
  suite + SPI factory, features packaged in the JAR" testkit whose adoption ergonomics the Java PoC
  copies.