Golden Vector Testing: Keeping Seven SDKs in Agreement

A user opens your checkout page. The Node service that renders it puts them in the new flow. The Go service that prices the cart puts them in the old one. Both services read the same flag, with the same 50% rollout, for the same user id, and neither one logged an error.

A feature flag vendor has to rule that failure out before anything else. In June 2026 we found five versions of it in our own SDKs, and we found every one by reading code. Featureflip’s server SDKs evaluate flags locally, and seven of them carry their own implementation of the evaluator, in JavaScript, Python, Go, Java, C#, PHP and Ruby. This post covers how we stopped relying on careful review to keep them in agreement. One engine generates a shared set of test cases, which we call golden vectors, and every SDK has to reproduce them in CI.

Key Takeaways

• An SDK’s own tests can only check that it agrees with itself. Parity across languages needs an oracle that sits outside every implementation.
• We generate the expected answers by running the production evaluation engine over a file of declared inputs, then copy the output into eight test suites, byte for byte.
• Three CI checks hold it together: a hash comparison of the eight copies, the engine re-checking its own fixture, and a harness in each SDK.
• Five of the nine vector classes are written by hand, because for those cases the engine can’t be the oracle.
• Vectors only catch what someone thought to write down, so each divergence you find gets a vector before its fix merges.

1. Why seven evaluators exist at all

Server SDKs evaluate locally. Your service downloads the flag configuration once, keeps it current over a streaming connection, and decides each flag in memory without a network call. If the connection drops, it keeps serving the last copy it received. That makes an evaluation cheap enough to run on every request, and it’s the usual design for server SDKs across flag platforms.

The cost is that the rules live in eight places. The Evaluation API has its own engine, written in C#. It answers the hosted HTTP endpoint, and it answers for the browser and mobile SDKs too, which receive results the server already worked out. Then each server SDK carries a port of that logic in its own language: rule matching, segments, prerequisite flags, condition operators, date parsing, semver comparison and percentage bucketing. The Node and Cloudflare Workers SDKs wrap the JavaScript core, so they inherit its answers, which still leaves seven ports where two languages can read one rule two ways.

2. Five divergences, all found by reading

In mid-June an audit of the engine against the SDKs turned up five bugs in a row, each one an input where some SDKs and the engine gave different answers.

Bucketing hashes salt:userId with MD5, reads the first four bytes as an unsigned little-endian integer and takes that modulo 100. Change the salt and the new bucket has nothing to do with the old one, which made the salt bug the worst of the five. On a 50/50 rollout, roughly half your users would get a different variation from a Go service than from a Node service.

Every SDK had tests in all five areas. All of them passed. The salt tests in Go, C# and Java compared two configurations against each other, or called the SDK’s own bucket() function to compute the expected value. Those tests would have passed with the argument order swapped. What was missing was any check that implementation A agrees with the engine on a known input. Testers call that the oracle problem, and adding more tests inside each SDK never touches it, because every one of them still asks the same SDK for the answer.

3. One engine writes the answers

The fix is a small .NET tool that references the real evaluation engine. It reads a file called vectors.inputs.json, which declares inputs only: a salt and a user id, or a condition and an attribute value, or a whole flag with its rules and a context. The tool runs each input through the engine and records what the engine says. Then it writes the result, vectors.json, into eight places: the engine’s own test project and the test tree of each server SDK.

A bucket vector is about as small as a test case gets:

{
  "id": "b-basic",
  "salt": "flag-abc",
  "value": "user-42",
  "expectedBucket": 46
}

Nobody typed the 46. The engine produced it, and now JavaScript, Python, Go, Java, C#, PHP and Ruby each have to hash flag-abc:user-42 and land on 46 too. Condition vectors pair an operator, an attribute value and a negate flag with an expected match. Flag vectors carry a full flag configuration with its segments and a context, and they expect a variation, a value and a reason.

The bottom row of the figure is what runs in CI, and each check catches a mistake the other two miss.

1. A sync job hashes the eight copies with SHA-256 and fails if it finds more than one distinct hash. It has no .NET toolchain and never regenerates anything. Its whole job is catching a copy someone edited by hand or left out of a commit.
2. The engine re-runs its own fixture. Change evaluation behavior without regenerating, and the engine’s test suite goes red. This is the check that catches engine drift. The sync job can’t, because eight stale copies still match each other.
3. Each SDK runs a harness over the fixture in its own CI, with its own test runner. An SDK that has wandered away from the engine fails here.

Reason strings needed one adapter. The engine reports rule-match in kebab case, the JavaScript core uses PascalCase and PHP uses screaming snake case, because each evaluator is the source of truth for its own runtime. The vectors store a structured reason, and each harness maps its own spelling onto that before comparing.

The suite started at 39 vectors in June. It holds 318 today, across nine classes. A new case goes into the inputs file and nowhere else. An assertion written straight into one SDK’s tests would be a second contract, and the next divergence would hide in the gap between the two.

4. When the engine can’t be the oracle

Four of the nine classes come straight from the engine: buckets, rollouts, conditions and whole flags. The other five are written by hand, expected answer included, and the generator copies them through untouched. We had to write down why each one couldn’t be generated, and the five reasons are different enough to need different fixtures.

Core contract: the engine disagrees. A typed accessor like boolVariation has to return the caller’s default when the stored variation isn’t a boolean. The engine sits a layer below that and returns null. Generating those cases would lock in the wrong answer, so the core contract vectors are written out by hand.

Unknown operators: the engine can’t express them. The generator turns operator names into a C# enum, which throws on a name it doesn’t know, so no generated vector could ever contain one. That blind spot hid a nasty bug. An SDK that met an operator it didn’t recognize returned “no match”, and a negated condition flipped that to “match”. A rule reading “NOT (some operator I don’t understand)” matched every user. The realistic trigger is an operator released on the server reaching an SDK pinned to an older version. The fix makes an unknown operator mean “can’t evaluate”, which negation doesn’t invert, and a hand-written class now holds the four SDKs that read operators as raw strings to it. The other three parse operators into an enum and reject an unknown one before evaluation starts.

Dates: the engine dissents on purpose. The engine parses dates with DateTimeOffset.TryParse, which accepts 05/15/2023 and Jan 1 2024. Every SDK accepts only ISO 8601. Making the engine stricter would change results for rules customers had already saved, so the leniency stays, the Management API refuses non-ISO dates when a rule is written, and 64 hand-written date vectors pin what the SDKs do.

Malformed config: the engine never sees it. One server bug sent enums as integers over the stream and as strings over REST. Seven SDKs met that payload. Between them they produced five different outcomes, and two of those were silently wrong. The engine would never produce such a payload, so there is nothing for it to compute, and the malformed-config vectors record a disposition instead: reject the whole snapshot and keep serving the last good one, or accept it.

Events: the engine has nothing to say. Evaluation returns a result and emits no events. The payload that identify() and track() send is built in each SDK’s shared core. In August we found that identify() built a different payload in three of the six SDKs that offered it. JavaScript, Node and Python sent the caller’s attributes while PHP, Go and Ruby dropped them, and it went unnoticed because nothing compared an emitted event against an expected shape, which is the comparison the event vectors now make.

5. Rules we picked up writing them

• Never write a vector over a divergence. When two SDKs genuinely disagree, a shared vector picks a winner by accident and locks it in. Operator spelling taught us this one. Go lowercased names without stripping underscores, so it accepted notequals and refused not_equals. PHP did the opposite. JavaScript and Ruby accepted only the exact PascalCase name, so what counted as a known operator depended on which language you asked. The spelling vectors waited until every SDK stripped underscores and folded case, which accepts everything any of the four accepted before. No SDK got stricter, and nothing that evaluated stopped evaluating.
• Assert the exact field set, on the serialized bytes. The identify() bug was a field present in some SDKs and absent in others, which a subset check can’t see. Absence also only exists after serialization: Go’s omitempty, Java’s NON_NULL and PHP encoding an empty array as [] are all invisible one layer up.
• Check an invariant where SDKs legitimately differ. Event timestamps come out four ways across the fleet. Python writes microseconds with +00:00, JavaScript and PHP write milliseconds with Z, Go and Ruby write whole seconds, and Java’s precision varies. The vectors check that the value parses as an ISO 8601 instant in UTC and leave the spelling alone.
• Count what ran. The runners for the hand-written classes assert a minimum number of executed vectors. A harness that silently skips everything still passes, which is worse than having no harness, because it looks like coverage.
• Skip by capability, never by SDK name. Some event vectors need a context that is an untyped map. Go and Java keep the user id in a dedicated field, so those cases can’t arise there. Each vector names the capability it needs. A runner skips what its SDK can’t express, and because the fixture records the reason for every skip, nobody can quietly widen one later into coverage it never had.

6. What vectors don’t catch

In October, while deciding what should happen to a context that can’t be bucketed, we found that a boolean user id hashed three different ways. The engine, Python and C# turned true into the string True, the default spelling in .NET and Python. JavaScript, Go, Java and Ruby used true. PHP used 1, with an empty string for false, and since the string going into the hash differed across those three groups, so did the bucket a user landed in. Numbers had the same problem: a user id written as 1.0 hashed as 1.0 in some languages and as 1 in others.

No vector caught either one, because no vector had a boolean or fractional user id. All seven SDKs passed the whole suite while disagreeing. We found both by reading code.

Each fix landed with vectors that pin it, so every evaluator now hashes booleans as lowercase true and false and spells fractional numbers the way JavaScript’s Number.prototype.toString does, and CI holds them there. In June a fix meant patching several SDKs and hoping nobody reintroduced the bug. The patching is the same today, plus one entry in the inputs file, and a regression fails the next pull request in whichever SDK it lands in.

7. Doing this in your own codebase

A serializer, a hashing scheme, a pricing rule or a parser shipped in several languages carries the same exposure as a flag evaluator. Here’s what we’d keep if we started over.

• Pick one implementation as the oracle and generate expectations from it, because hand-computed expected values drift from the real code the day someone changes it.
• Use plain JSON. Every language you ship can parse it with its standard library.
• Copy the fixture into each package and check the copies are identical. Each package’s CI then runs on its own, with no shared path to resolve.
• Write down why each hand-written class can’t be generated. “The oracle disagrees”, “the oracle can’t express it” and “the oracle has no opinion” lead to different fixtures, and mixing them up gets you vectors that lock in bugs.
• Keep auditing by reading. Both of our big rounds of divergences, in June and in October, came from people reading code.

If you want the user-facing side of this, why two SDKs can return different variations covers what to check when two of your services disagree, and rollout strategies documents how bucketing works for keyed and keyless contexts.

The shorter version

Seven server SDKs evaluating flags locally means seven chances to read a rule differently, and per-SDK tests can’t see it, because each one only checks itself. So the production engine answers a file of declared inputs, and every SDK’s test suite gets a copy of those answers. Three CI checks keep the copies, the engine and the SDKs in line. The engine can’t be the oracle for five classes. Those are hand-written. New divergences still turn up by reading code, and each one now ships with a vector that keeps it fixed.

Frequently asked questions

What are golden vectors in testing?

Golden vector testing uses a fixed set of inputs paired with expected outputs, saved as a file in the repository. A test runs each input through the code and compares the result with the saved output. It’s golden master testing applied across implementations: several languages run the same file, so they all have to produce the same answers.

Why do feature flag SDKs need cross-language tests?

Server SDKs evaluate flags inside your process for speed, so each language carries its own copy of the targeting logic. If two copies read a rule differently, a user can get one variation from your Go service and another from your Node service. Tests written inside each SDK can’t detect that, because they only check the SDK against itself.

Does every Featureflip SDK use the same bucketing hash?

Yes. Every server SDK and the Evaluation API hash the flag’s salt and the user’s bucketing value with MD5, read the first four bytes as an unsigned little-endian integer and take it modulo 100. Shared vectors check the hash and also the step before it, where a boolean or a number is turned into the string that gets hashed.

What do golden vectors not catch?

Anything nobody wrote a vector for. Divergences still have to be found, by reading code or from a bug report. Once one is found, the fix ships with vectors, and an SDK that drifts back fails CI.

Do the browser and mobile SDKs run the vectors?

They don’t need to. The browser, React, Swift, Android and Flutter SDKs receive results the Evaluation API has already computed, and that engine runs the full vector set in its own test suite.

Featureflip is a flat-priced feature flag platform with 14 official SDKs, and every server SDK is held to one shared set of evaluation vectors in CI. Start on the Free plan without a credit card.