An analytics concept defining statistical methods used to estimate relationships, evaluate interventions, and generate forecasts. It specifies data requirements, estimation procedures, and uncertainty measures used to support decision-making. It does not prove causation without an appropriate identification strategy, data quality checks, and sensitivity analysis. It supports performance management by translating data into estimates, predictions, and quantified uncertainty. The concept is generally stable, though tooling and best practices for measurement evolve over time.

Principle

Random assignment ensures exchangeability between variants so that differences in outcome can be attributed to the treatment; pre-specify metrics, sample size, and analysis plan to control type I/II errors and avoid data-driven peeking.

Demonstration

A product team randomizes users to see either the existing checkout flow (A) or a simplified checkout (B) and measures conversion rate over a pre-specified period; analysis estimates the average treatment effect on conversion with confidence intervals and pre-registered stopping rules.

Misapplication

Peeking at results repeatedly and stopping when statistically significant (optional stopping), failing to account for multiple comparisons when testing many variants, or non-random assignment of users to variants (e.g., by device type) that invalidates causal inference.

Consequence

When properly randomized and analyzed, A/B tests produce credible causal estimates of feature impact, enabling data-driven product decisions, prioritized rollouts, and measurement of incremental value versus status quo.

Reversal

Comparing cohorts that self-select into different experiences or using observational before/after comparisons without randomization yields associational evidence that may reflect confounding, selection, or time trends rather than treatment effects.

Boundary

A/B testing assumes stable units and no interference between units (SUTVA) and sufficient sample size; not appropriate when effects are rare, when changes cause network spillovers, or when ethical/legal constraints prevent randomization.

Semantic Tension

Sometimes conflated with multivariate or factorial experiments and with bandit/adaptive testing; A/B tests are simple randomized contrasts, while bandits optimize allocation adaptively and factorial designs estimate interaction effects across multiple factors.

Synthesis

A/B testing is the basic randomized experiment in product and business settings: randomize comparable units into variants, measure pre-specified outcomes, and analyze under the randomization framework to obtain causal estimates that guide deployment and business strategy.