Experimentation

Experimentation needs three things: a way to assign users to variations, a way to keep that assignment stable, and a way to measure outcomes. Feature flags supply the first two (a flag serves each user a variation, and a deterministic hash keeps them in their group for the duration), so the same flag that controls a release can also hold a clean experimental split. The measurement itself comes from the metrics you already track against each variation.

They overlap but optimize for different goals. Feature flags and the feature management layer around them handle release, which means decoupling deploy from launch, ramping safely and killing bad code. Experimentation is about deciding which version to keep, based on measured results. The same flag can serve both: ship a feature gradually and measure its effect on the metric you care about.

Four properties separate an experiment from a change you happened to measure afterwards. Assignment has to be random, so that the groups differ in the treatment and in nothing else that matters. The metric and the effect size have to be fixed before the test starts, because choosing them once the data is in lets you find a winner in noise. The sample size follows from that effect size, and it decides how long the test runs. Fix the stopping point in advance as well. Checking every morning and stopping the first time the line crosses is the most common way a team talks itself into a result that will not reproduce.

Underpowered tests are the usual reason, and the arithmetic behind it is unforgiving. Detecting a one percent lift takes far more traffic than detecting a ten percent one, and most product changes move the metric by small amounts. A test that ends with no significant difference has still told you something true, which is that the effect is smaller than the size you powered for. The expensive version is the test that was never large enough to detect the effect you were hoping for, because it costs a release cycle and returns nothing either way. Work out the required sample before you build the variation. When the number is out of reach, ship the change behind a percentage rollout and watch your error rates and flag metrics for regressions instead of claiming a lift you cannot measure.