Quick answer
An Overall Evaluation Criterion, or OEC, is the pre-agreed metric or metric function used to decide whether one experiment variant is better than another. It translates a long-term objective into a measurable short-term criterion and reduces post-hoc metric shopping. A good OEC has a clear population, event definition, denominator, direction, horizon and economic or customer interpretation. It responds to meaningful treatment effects, is difficult to game, and has evidence linking it to longer-term value. It may be a single outcome or a documented composite, but weights and units must be governed rather than tuned after results. Data-quality metrics establish trustworthiness, diagnostics explain mechanisms, and guardrails prevent unacceptable harm; they do not all become equal success criteria. Because short-term proxies can mislead, organizations should validate the OEC against longer-term cohorts and experiments, review it when strategy changes, and report uncertainty and tradeoffs rather than forcing every decision into one number.
What is an Overall Evaluation Criterion?
The Overall Evaluation Criterion is the metric or metric function designated to evaluate variants in a controlled experiment. It provides a shared answer to what better means before teams see which available metrics moved.
The term is associated with large-scale online experimentation, where products can generate thousands of measures but decisions still need a coherent criterion. An OEC may be conversion, retained use, contribution or a composite designed to approximate longer-term value.
Overall does not mean the number perfectly captures every consequence. It means the organization has made its principal tradeoff explicit. Guardrails, diagnostics, qualitative evidence and judgment still matter around that criterion.
Bridge short experiments to long-term value
Many decisions concern outcomes that mature over months or years, while experiments run for days or weeks. The OEC must therefore respond within the study while preserving a credible relationship to retention, trust, customer benefit or economic value over the longer horizon.
Short-term metrics can point the wrong way. More searches may indicate engagement or poor results. More notifications can raise opens while increasing fatigue. Higher immediate revenue can come from pricing or pressure that damages later retention.
Microsoft research describes the ideal OEC as short-term metrics that predict long-term value and documents pitfalls in long-running experiments, including identifier stability, survivorship and selection. Neither short nor long windows remove the need for careful validation.
Design the OEC from value to decision
Start with the beneficiary, desired long-term outcome and mechanism. Then write the experimental estimand and a precise metric specification. Only after that should the team evaluate historical behaviour, sensitivity and operational feasibility.
Create a metric tree that connects the OEC to diagnostics and guardrails. The tree should show how treatment could influence each element and where a positive OEC might conceal harm. This makes tradeoffs debatable before a launch emergency.
Version the OEC, query, data sources, owner, validation evidence and decision policy. If a definition changes, do not splice old and new results into one trend without a bridge analysis.
Long-term value
State whose value matters, over what horizon and through which customer mechanism.
- What outcome should improve in the long run?
- Which short-term behavior plausibly predicts it?
Metric specification
Define population, numerator, denominator, direction, horizon, transformations and missing-data rules.
- Can two analysts reproduce the metric?
- How could treatment change eligibility or the denominator?
Validation
Test sensitivity, causal responsiveness and the relationship to longer-term outcomes.
- Does the metric move when real value moves?
- Can it be gamed or distorted?
Guardrails and diagnostics
Separate the deciding criterion from trust, safety and mechanism metrics.
- What harm blocks shipping?
- Which metrics explain the result without redefining success?
Decision and review
Predefine tradeoff rules, apply uncertainty and revise the OEC when its evidence or strategy changes.
- What effect is worth acting on?
- When does this metric need revalidation?
Specify the metric so it can be reproduced
Name the analysis unit and eligible population. Define numerator, denominator, event qualification, attribution to the unit, observation window, time zone, duplicate handling, missingness, late arrival and whether values are capped or transformed.
Denominators deserve special attention. A treatment can change who reaches a page or triggers a metric, making a per-triggered-user result subject to selection. Prefer assignment-based populations for the primary causal comparison unless the estimand explicitly justifies another denominator.
For ratios and composites, specify how aggregation occurs. The average of user-level ratios is not always the ratio of totals. Weights, normalization and currency conversion can change the result and should never be left to dashboard defaults.
Use composite criteria with discipline
A composite combines several measures when no single outcome represents the objective. It can balance quantity and quality, or near-term behaviour and economic value. The benefit is an explicit tradeoff; the danger is hiding judgment inside weights and transformations.
Choose components from a causal and strategic model, place them on interpretable scales and test sensitivity to plausible weights. Avoid fitting weights to make past favored decisions look correct. If reasonable weight choices reverse the conclusion, report the fragility rather than one precise score.
Keep the composite understandable enough for teams to predict how product changes influence it. An opaque criterion can be gamed unintentionally and makes debugging difficult. Sometimes a primary metric plus hard guardrails is more governable than a weighted index.
Validate sensitivity and long-term meaning
Use historical cohorts to test whether the early metric predicts later outcomes, while remembering prediction is not causation. Study known beneficial and harmful changes, A/A tests, randomized long-term holdouts and natural variation to identify counterexamples.
Assess sensitivity to event definition, horizon, caps, bot filtering, missing data and population. Stable decisions across reasonable specifications increase confidence. A metric that flips under small arbitrary changes is a warning.
Revalidate when the product, customer base, business model, instrumentation or strategy changes. Once teams optimize a metric, behaviour adapts and its relationship to value can deteriorate, a practical form of Goodhart's law.
OEC example
The language-learning app replaces a count that rewards repeated taps with a criterion tied more closely to retained learning time. It still treats the proposed metric as a hypothesis requiring long-term validation, not as a perfect measure of education.
Metric roles are explicit. Diagnostics explain the lesson funnel, guardrails protect learners and data-quality checks determine validity. None is selected as the winner simply because it moved favorably.
A hypothetical language-learning app currently rewards teams for increasing daily lesson completions, even though very short repeated lessons can raise the count without improving durable learning or subscription value.
The team states the long-term aim as sustained learner progress and acceptable subscriber contribution. It studies which early behaviours predict retained mastery and renewal without causing frustration.
The proposed OEC is retained mastery minutes per eligible learner over a fixed follow-up window, with a cap to reduce domination by a few extreme users. The exact calculation and late-event handling are versioned.
Lesson starts and completion funnel are diagnostics. Crash rate, latency, refund requests, accessibility failures and unhealthy-use indicators are guardrails. SRM and telemetry checks determine whether results are trustworthy.
Historical cohorts and long-running holdouts test whether changes in the short-term criterion predict later retained learning and renewal. The team checks segments and known product changes for counterexamples.
Experiments ship only when the OEC interval clears a practical threshold and guardrails remain acceptable. Exceptions require an explicit review rather than replacing the OEC after results arrive.
This example is hypothetical. Composite learning or value metrics require domain validation and should not be interpreted as established educational measures.
Separate OEC, guardrail, diagnostic and data quality
The OEC makes the principal value judgment. Guardrails constrain unacceptable regressions in safety, reliability, fairness, customer experience or economics. Diagnostics illuminate the mechanism and help teams improve a treatment even when it does not ship.
Data-quality metrics such as sample ratio, logging coverage and invariant checks determine whether the experiment can be trusted. They are not treatment benefits. A large OEC gain cannot compensate for unexplained SRM or treatment-specific missing data.
Predefine how conflicts are resolved. A positive OEC with a guardrail breach may be rejected, revised or escalated according to severity and uncertainty. An exception should record the rationale and owner rather than silently redefining the criterion.
Use the OEC across experiments without losing context
A shared OEC lets teams compare changes and accumulate learning, but different surfaces may need specialized estimands or guardrails. A billing experiment and a search-ranking experiment can contribute to long-term value through distinct mechanisms.
Monitor interactions among simultaneously shipped changes. Each may improve the OEC alone while their combined effect saturates, conflicts or shifts customer mix. Portfolio reviews and occasional combined tests can reveal these system effects.
Track the distribution of experiment effects, replication and post-launch performance. If many wins disappear at rollout or fail to predict business outcomes, audit the OEC and experiment process rather than demanding more tests.
Limitations and common mistakes
Any proxy is incomplete. Rare severe harms, brand trust, accessibility and effects outside the measured platform may not appear in the OEC. Qualitative research, expert review and longer-term monitoring remain necessary.
Common mistakes include choosing the metric after results, using a vanity activity count, changing denominators, hiding weights in a composite, declaring diagnostics to be wins, ignoring guardrail uncertainty and assuming historic correlation guarantees future value.
An OEC can centralize power over what the organization values. Include product, finance, customer, data, legal and risk perspectives where relevant. The metric should make tradeoffs visible, not make stakeholder judgment disappear.
The OEC prevents metric shopping only when its definition, evidence and tradeoffs are fixed before the result is known.
Frequently asked questions
Is an OEC the same as a primary metric?
Often it is the primary decision metric, but the term emphasizes an organization-wide criterion designed to represent overall and longer-term value across experiments.
Can an OEC contain several metrics?
Yes, but a composite needs transparent components, units, weights, sensitivity analysis and governance. A single primary metric plus guardrails can be simpler.
What is the difference between an OEC and a guardrail?
The OEC defines the principal improvement. A guardrail sets a boundary against unacceptable harm or degradation.
How do I know whether a short-term OEC predicts long-term value?
Use longer-term cohorts, randomized holdouts, known interventions and sensitivity checks, then revalidate as the product and population change.
Can the OEC be changed?
Yes, when strategy or evidence changes, but version it prospectively. Do not change it for a running experiment merely because another metric looks better.
Sources and further reading
- Microsoft Research: Pitfalls of Long-Term Online Controlled Experiments ↗Primary research defining the OEC challenge and pitfalls in relating short experiments to long-term value
- Microsoft Research: Patterns of Trustworthy Experimentation During Experiment ↗Practitioner taxonomy for data-quality, OEC, diagnostic and guardrail metrics
- Cambridge University Press: Trustworthy Online Controlled Experiments ↗Authoritative practical reference on OEC design and trustworthy decision making
- Microsoft Research: Top Challenges from the First Practical Online Controlled Experiments Summit ↗Primary industry research on OEC, long-term value and experimentation challenges