Quick answer
An A/B test randomly assigns eligible experimental units, such as users, accounts, sessions, stores or geographic markets, to a control condition A and a treatment condition B. If assignment is implemented correctly and the conditions differ only as intended, the outcome difference estimates the treatment's causal effect for that tested population and period. A trustworthy test starts with a decision, hypothesis, primary metric, guardrails, minimum detectable effect, unit, duration and analysis plan. It then checks assignment integrity, especially sample ratio mismatch, verifies treatment delivery, accounts for conversion lag and reports effect size with uncertainty. Repeated peeking, changing metrics after results, conditioning on post-treatment behaviour, ignoring interference or shipping from a statistically significant but trivial movement can all produce bad decisions. One test supports a bounded claim, not a universal truth.
What is an A/B test?
An A/B test is a randomized controlled comparison between a baseline and a treatment. Eligible experimental units are assigned by chance before the outcome, which makes the groups comparable in expectation and supports a causal estimate of the treatment difference.
The unit may be a person, account, household, device, session, store, campaign or market. The correct choice follows how treatment is delivered and where interference occurs. Randomizing sessions is unsafe when a returning person can see both variants and earlier exposure changes later behaviour.
A/B testing is not merely showing two versions and choosing the larger conversion rate. Without valid assignment, reliable metrics, adequate power and a planned analysis, the comparison can be more misleading than an honest observational report.
Start with a decision, not a dashboard
Write the decision the test will support: ship, reject, revise, increase a budget, change a message or learn about a mechanism. Define the exact treatment and control, eligible population, time horizon and minimum effect that would justify action.
A useful hypothesis connects change to mechanism and outcome. Showing pickup windows earlier should reduce uncertainty and increase collected orders. The mechanism can be checked with diagnostics, but success should rest on the customer or business outcome.
Record what the test cannot answer. A checkout test on existing members may not generalize to new customers, mobile apps, another country or a holiday peak. Bounded claims make experimentation cumulative rather than theatrical.
Use a decision-to-validation framework
Predeclare the primary metric, guardrails, important diagnostics, analysis unit, allocation, eligibility, exclusion rules, duration and stopping policy. This reduces the freedom to redefine success after seeing results.
Keep one primary decision criterion unless a documented composite is necessary. Multiple guardrails can protect reliability, safety, customer experience and economics. Diagnostics explain the path but should not quietly replace a failed primary outcome.
Assign experiment, analysis and decision owners. Preserve configuration, code version, launch time, ramp changes, outages and query definitions. A result that cannot be reproduced from a dated protocol is hard to trust or reuse.
Decision and hypothesis
State the change, mechanism, population and decision the result will inform.
- What exactly differs in B?
- Why should that difference move the outcome?
Metrics and guardrails
Predeclare one primary outcome plus quality, safety and data-integrity checks.
- Which metric represents value?
- What must not worsen?
Design and power
Choose the randomization unit, allocation, minimum effect and duration before launch.
- Can units interfere?
- Can the sample detect a decision-relevant effect?
Run and validate
Protect assignment and verify sample ratio, exposure, instrumentation and concurrent changes.
- Did traffic split as configured?
- Was treatment delivered consistently?
Estimate and decide
Report the causal contrast with uncertainty, practical importance and a bounded action.
- How wide is the plausible effect?
- Does it clear the predeclared threshold?
Plan power around a meaningful effect
Power is the probability that the design detects an effect of a specified size when that effect exists. It depends on baseline rate, variability, sample size, allocation, metric shape, assignment clustering and the chosen false-positive and false-negative tolerances.
Choose a minimum detectable effect from decision economics, not from the sample available this week. If the test can detect only a very large change, a non-significant result cannot rule out smaller but valuable effects. If it detects tiny changes, statistical significance can still be commercially trivial.
Estimate duration from eligible traffic and outcome lag, then include complete weekly cycles or known seasonality where relevant. Do not repeatedly extend a fixed-horizon test until the p-value crosses a preferred boundary. Sequential methods can support valid continuous monitoring when selected in advance.
Choose and protect the randomization unit
Randomize at the lowest unit that can receive one stable condition without contaminating others. Account-level assignment can preserve consistency across devices when login is reliable. Cluster assignment may be needed for stores, sales teams or communities, but fewer independent clusters reduce power.
Use deterministic persistent assignment so the same unit remains in its variant. Stratification can improve balance for important pre-treatment attributes, while triggered analysis can improve relevance only if triggering is defined identically and not caused differently by treatment.
Analyze units according to assignment for the primary intent-to-treat estimate. Comparing only people who clicked, saw a component or completed a treatment-specific event conditions on post-treatment behaviour and can destroy the benefit of randomization.
Run trustworthiness checks before effects
Sample ratio mismatch occurs when observed allocation differs unexpectedly from the configured ratio. It can signal broken identifiers, assignment, redirects, logging, joins, filters or self-selection. Microsoft research treats SRM as a diagnostic symptom that must be investigated, not waived because the result looks plausible.
Also check invariant pre-treatment attributes, missingness, event volumes, duplicate records, exposure, page errors and latency. Confirm that the treatment actually differed as designed and that control did not receive it through caching, sharing or another campaign.
Document concurrent releases, promotions, outages and external shocks. Randomization usually balances common shocks, but interactions or differential implementation can still alter interpretation. Pause or classify the result as invalid when integrity cannot be restored.
A/B testing example
The food cooperative uses collected orders rather than checkout clicks because the change could shift uncollectable or refunded demand. Persistent account assignment also prevents a shopper from learning one pickup interface and being measured in the other.
The team reads SRM, errors and opening-hour changes before the outcome. A surprising improvement is evidence only after the experiment chain is shown to be trustworthy.
A hypothetical community food cooperative suspects that shoppers abandon online orders because available pickup times appear only after they enter payment details.
The treatment shows actual pickup windows before payment. The proposed mechanism is reduced uncertainty, and the primary outcome is the share of eligible checkout starters who place and later collect a paid order.
Refunds, missed collections, support contacts, basket contribution and page performance are guardrails. A click on the window selector is diagnostic, not the deciding metric.
Eligible customer accounts are assigned persistently to A or B. The team sizes the test for the smallest collected-order improvement worth shipping and includes the normal pickup and cancellation lag.
Before reading the treatment effect, the analyst checks sample ratio, eligibility counts, event completeness, page errors and whether any store changed opening hours unevenly.
The team reports the absolute difference and interval. It ships only if the plausible collected-order gain is material and guardrails remain acceptable, then monitors the post-launch result.
This example is hypothetical. A real analysis must use methods appropriate to the assignment unit, metric distribution and decision policy.
Report effect size and uncertainty
Show the treatment and control outcomes, absolute difference, relative difference and a confidence or credible interval using a method appropriate to the metric and assignment. Explain the interval as a range generated by the stated procedure, not as certainty that every value is equally likely.
Separate statistical significance from practical significance. A narrow positive effect below implementation cost may not justify shipping. A wide interval spanning meaningful harm and benefit is inconclusive, even if the point estimate is positive.
Correct or constrain multiple testing when many variants, metrics or segments can trigger a claim. Label exploratory subgroup findings and replicate important heterogeneity. Do not present the most favorable slice as the original experiment conclusion.
Account for novelty, interference and carryover
Novelty can create a temporary response because a treatment is unfamiliar, while learning can make an initially difficult design improve. Plot effects over preplanned time bands cautiously and run long enough to cover the decision-relevant behaviour without searching for a favorable window.
Interference occurs when one unit's treatment affects another, such as marketplace pricing, referrals, shared households or inventory constraints. The usual independent-unit estimate may no longer represent the intended effect. Cluster, geo or switchback designs may better match the mechanism.
Carryover matters when units experience both conditions over time. Washout can help only when effects truly decay. For durable learning, subscription or brand changes, parallel assignment is usually easier to interpret than rapid crossovers.
Limitations and common mistakes
Randomization estimates the effect of the tested implementation for the tested eligible population. It does not explain every mechanism, guarantee future performance or settle effects at a different scale. Large rollouts can change queues, auctions, social interaction or customer mix.
Common mistakes include testing several substantive changes under one label, peeking without correction, changing the primary metric, ignoring SRM, analyzing exposed users only, ending before conversion lag, and calling non-significance proof of no effect.
Some questions cannot be randomized ethically or practically. Use a different causal or qualitative method and state its assumptions. The prestige of an experiment should never override consent, safety, accessibility or fair treatment.
Randomization earns a causal comparison only when assignment, measurement and analysis preserve the comparison it created.
Frequently asked questions
How long should an A/B test run?
Run for the preplanned duration required by eligible traffic, power, business cycles and outcome lag. There is no universal one-week or two-week rule.
What is sample ratio mismatch?
SRM is an unexpected difference between configured and observed variant allocation. It often signals a data or experiment integrity problem that must be diagnosed before effect analysis.
Can I stop when a test becomes significant?
Not under an ordinary fixed-horizon analysis. Use a preselected valid sequential method if continuous monitoring and early stopping are operational requirements.
Does a non-significant result mean no effect?
No. It means the data did not meet the chosen evidence threshold. Inspect the effect interval to see which beneficial or harmful effects remain plausible.
Should I test more than one metric?
Use one primary decision criterion plus predeclared guardrails and diagnostics. Adjust interpretation when many metrics or segments can generate a success claim.
Sources and further reading
- Cambridge University Press: Trustworthy Online Controlled Experiments ↗Authoritative practical reference for online randomized experiments, metrics, power and trustworthiness
- Microsoft Research: Diagnosing Sample Ratio Mismatch ↗Primary research on SRM causes, diagnosis and its threat to trustworthy decisions
- Microsoft Research: Patterns of Trustworthy Experimentation During Experiment ↗Practitioner guidance on experiment monitoring, data quality and execution
- NIST: Comparisons Based on Data from One Process ↗Statistical reference for comparison procedures, assumptions and uncertainty