Holdouts and Counterfactuals: Measure What Would Have Happened Anyway

What counterfactuals and holdouts mean

For an eligible customer, campaign or market, a counterfactual outcome is what would have happened under a different treatment condition. Causal effect is the contrast between potential outcomes, but only one can be observed for the same unit in the same period. This is the fundamental missing-data problem.

A holdout is a group deliberately kept from an intervention or kept on a defined baseline. If assignment is randomized and the design remains intact, the control group's average outcome estimates the missing untreated outcome for the treated group, subject to sampling uncertainty and assumptions about interference.

The question is always compared with what. A business-as-usual control estimates the effect of replacing current practice with the new program. A no-marketing control estimates a broader effect. These are different estimands and can produce different answers without contradiction.

Understand the causal logic

Randomization makes treatment assignment independent of pre-existing potential outcomes in expectation. With enough units and correct execution, observed differences after assignment can be attributed to the offered treatment rather than customer intent, seasonality or the targeting model.

Three ideas still matter. Consistency requires that the named treatment corresponds to what units actually experience. Positivity requires a genuine chance of receiving each condition for the eligible population. Limited interference requires one unit's assignment not to materially change another unit's outcome, unless the design models that connection.

A holdout does not need both groups to look identical on every observed variable. Random differences occur. Pre-period balance and sample-ratio checks diagnose implementation, while the analysis uses uncertainty rather than deleting inconvenient units until the groups look favorable.

Design a credible holdout

Write the estimand before assignment: eligible population, unit, treatment, baseline, outcome, window and aggregation. Choose the randomization unit to match delivery and interference. Customer-level assignment suits direct messaging; household, store, market or geographic clusters may be safer when exposure spills across people.

Estimate the required sample using the minimum effect worth acting on, baseline variation, allocation ratio, clustering and planned variance reduction. A smaller control preserves more treatment opportunity but can sharply reduce precision. Long-term outcomes require enough duration and stable assignment.

Stratify on important pre-treatment factors when useful, then lock a reproducible assignment. Suppression must cover every treatment channel in scope. Keep service, legal and safety messages outside the marketing treatment and document them equally for both arms.

Preserve assignment and diagnose integrity

Store assignment in a durable system, not in a campaign export. Downstream email, ad-audience, app and call-center tools should read the same state. Log attempted and delivered exposures so contamination can be measured without redefining groups after the fact.

Check sample-ratio mismatch against the planned allocation. Investigate eligibility filters, bot removal, missing outcomes, delayed joins and channel failures that affect arms differently. A statistically significant ratio mismatch is a warning about the experiment pipeline, not a nuisance to explain away.

Monitor spillover. People may share offers, sellers may change behavior when some accounts are suppressed, or media treatment in one region may reach another. Cluster assignment, buffer regions or exposure measurement can reduce problems, but each changes power and interpretation.

• Decision and estimand written
• Baseline condition explicit
• Eligible population frozen
• Randomization unit matches delivery
• Interference assessed
• Power and duration planned
• Assignment persisted
• All scoped channels honor suppression
• SRM and balance checked
• ITT analysis predeclared
• Guardrails and cost included
• Ethical exclusions documented

Estimate impact by intention to treat

The primary analysis compares units according to original assignment, whether or not the treatment was successfully delivered. This intention-to-treat estimate preserves the random comparison and answers the operational effect of offering the program under real execution.

Report the absolute difference, relative lift where interpretable, confidence or credible interval, sample size and outcome definition. A non-significant result is not proof of zero effect. State the range of effects consistent with the data and whether that range changes the decision.

Pre-period covariates or methods such as CUPED can improve precision when chosen before outcome inspection. Heterogeneous-effect analysis should be limited to prespecified, decision-relevant groups or treated as exploratory and validated in a new test.

Holdout and counterfactual example

The meal-kit company holds out the entire coordinated program, so its comparison estimates the combined incremental value of email, app, paid suppression and call-center activity. It does not allocate the effect among those channels. Separate factorial or channel tests would answer that narrower question.

Net contribution, not reactivation alone, governs the decision because discounts and service cost can make a higher return rate uneconomic. Unsubscribes and complaints protect customer trust. Persisted assignment prevents a customer from entering treatment through a different device or campaign.

Use universal and long-term holdouts carefully

A universal holdout withholds a broad portfolio of eligible marketing activity from a small stable group. It can estimate whether the combined program adds value beyond organic demand and expose double-counting across channel reports. It is particularly useful when many campaigns overlap.

Its result is broad but not diagnostic. A positive portfolio effect does not show which campaign caused it, while a near-zero total can combine positive and negative components. Pair the universal holdout with nested experiments or a structured learning plan where systems allow.

Long-term holdouts can reveal delayed retention, habit or brand effects, but carry larger opportunity and fairness costs. Reassess size, duration and eligibility, rotate where scientifically defensible, and stop when withholding would create material customer harm.

When random holdouts are difficult

Some treatments cannot be withheld, have nationwide spillover or involve too few units. Randomized geo experiments, switchback designs or phased rollouts may create a credible comparison. Each requires design around interference, time trends and treatment implementation.

Quasi-experimental methods can estimate counterfactuals from observational data using matched controls, synthetic controls, regression discontinuities or difference-in-differences. Their credibility depends on stronger assumptions than randomization, such as parallel trends or no unmeasured confounding.

State those assumptions and test observable implications. A sophisticated model is not an automatic substitute for design. If evidence remains weak, narrow the claim, reduce the decision size or invest in creating better variation.

Account for ethics, privacy and opportunity cost

Do not withhold safety, contractual, accessibility or essential service communication. Review whether customers in the control receive the normal standard of care, whether the intervention itself is appropriate and whether protected groups bear unequal cost.

Use the minimum personal data needed for assignment and analysis, restrict access, set retention and respect consent and platform rules. Pseudonymous assignment can reduce exposure, but it does not remove governance obligations when identities can be linked.

Quantify opportunity cost before launch using the largest plausible foregone value, not only the expected effect. Senior approval may be appropriate for large or long holdouts. The cost buys causal learning, so archive and reuse that learning across future decisions.

Limitations and common mistakes

Holdouts estimate the effect for the tested treatment, population, period and execution. Results may not transport to a different offer, market, season or scale. Treatment effects can change when a program moves from a limited test to saturation.

Common mistakes include analyzing only exposed users, allowing controls into other campaigns, ending the test after a favorable day, using too little power, ignoring spillover and reporting a relative lift without its base rate or interval.

A holdout also cannot explain mechanism by itself. Combine causal estimates with journey diagnostics and research. The goal is not to run a control group forever, but to make consequential decisions with a defensible comparison and a proportionate cost of learning.

Every incremental claim should complete the sentence: compared with what would have happened to the same eligible population, under which alternative, over what period?

Frequently asked questions

Sources and further reading

• Hernán and Robins: Causal Inference, What If ↗Primary textbook treatment of counterfactual outcomes, consistency, exchangeability and positivity
• Microsoft ExP: Patterns of Trustworthy Experimentation, Pre-Experiment ↗Primary practitioner guidance on controls, counterfactual logging and safe random rollout
• Google Research: Measuring Ad Effectiveness Using Geo Experiments ↗Primary research on randomized geographic treatment and control for advertising impact
• Microsoft Research: External Validity of Online Experiments ↗Research guidance on extended-duration experiments, reverse flights and holdouts for future effects

Explore related concepts