A/B testing

Also called: split testing

Last updated:

An A/B test is a controlled experiment that compares two variations — A, the control, and B, the treatment — by randomly assigning users to one or the other and measuring a target metric such as conversion or retention. Because assignment is random and the groups run concurrently, a statistically significant difference can be attributed to the change itself rather than to seasonality or audience mix.

How feature flags run an A/B test

A multivariate flag is the delivery mechanism: it serves variation A to one half of users and B to the other, assigning each user by a deterministic hash so they stay in the same group for the life of the test — the same percentage rollout bucketing, split for measurement rather than for a gradual ramp. The flag decides who sees which variation, and a metrics pipeline records how each group behaves so you can compare them.

A/B test vs percentage rollout

They share the bucketing machinery but answer different questions. A percentage rollout asks "is this safe to expand?" and ramps one variation toward 100%. An A/B test asks "which variation is better?" and holds a stable split long enough to gather a significant result. A/B testing is the simplest form of experimentation; richer designs test more than two variants at once.

Want the full picture? Read the concept guide: Rollout strategies →

Related terms

Experimentation

Experimentation is the practice of making product decisions from controlled experiments — running A/B and multivariate tests, measuring outcomes against a hypothesis, and shipping the version the data supports rather than the one with the loudest advocate.

Percentage rollout

A percentage rollout serves a feature to a defined share of users — chosen by a deterministic hash — so the same users stay in the cohort as you ramp the percentage up.

Feature flag

A feature flag is a conditional in your code whose value is controlled from a remote dashboard, letting you turn functionality on or off without deploying new code.

Browse the full feature flag glossary →

Try it in your own app

Free Solo plan covers 10 flags and 2 environments. No credit card, no demo call — sign up and ship.

Get started free See pricing