Incrementality: estimating what would not have happened without the action.
Directly verifiable evidence
Primary sources to check the key claims.
- Lewis and Rao (2015), Quarterly Journal of Economics (opens in a new tab)
Power and economic limits of advertising experiments.
- Holland (1986), Statistics and Causal Inference (opens in a new tab)
Causal framework and the distinction between association and effect.
Evidence status
What was not reproduced.
The source's power and sample-size calculations were re-run without substitution: 6 of the 8 values it publishes are reproduced to the digit, the other 2 within 1.2% and 1.9% — a gap of the order of the tolerance its own method declares, on a calculation that involves randomness. Its campaign data, its figures and its empirical results, by contrast, were never published: they cannot be re-run, by us or by anyone.
Key distinctions
Three conditions before calculating.
Estimand before method
Define outcome, population, treatment, comparator and horizon. "Measuring the impact" is not enough: average effect on the reachable, effect of the national budget and effect on new customers are three different quantities.
Credible absence of action
A before-after trend is a counterfactual only if nothing else changes the result. Seasonality, selection of the exposed and simultaneous campaigns must be neutralised or modelled.
Use limited by the evidence
A local test can authorise a similar rollout, but not necessarily an annual allocation. Extrapolation must be separated from identification.
Method
Building a measurement that authorises a decision
- 01Write the counterfactual
Describe concretely what happens to the unit without the action — no message, usual budget, previous price or another channel.
- 02Choose the unit of assignment
Individual, store, area or period, according to the risk of contamination and the level of execution.
- 03Size the uncertainty
Compute the minimum detectable effect and check that the decision stays useful even if the interval is wide.
- 04Pre-register the use
Set metric, exclusions, duration, analysis and decision rule before observing the results.
The matrix of counterfactuals
Four families of methods are compared on what they identify, their assumptions and the decisions they can genuinely support.
| Method | Granularity | Counterfactual | Identification | Suitable decision |
|---|---|---|---|---|
| Individual test | Individual | Randomised | Very high | Targeted activation |
| Geo test | Area | Comparable areas | High | Aggregate media |
| Difference in differences | Unit × time | Parallel trends | Conditional | Price or channel |
| Synthetic control | Market | Combination of controls | Conditional | Local rollout |
| MMM | Week / area | Modelled structure | Depends on the model | Annual portfolio |
It is the one whose counterfactual, granularity and horizon match the real decision.
An attributed ROAS of 4 for an incremental ROAS of 1.3
A campaign spends €100,000 and the platform attributes €400,000 of sales. A comparable unexposed group would have generated €270,000 on the same population.
Incremental sales: 400,000 − 270,000 = €130,000. Incremental ROAS: 1.3. With 45% gross margin, the contribution before creative costs is only €58,500, below the spend.
The campaign captures existing sales. Reduce or redefine the target rather than amplify it on the basis of the attributed ROAS.
Acceptance conditions
What must be true to act.
- 01
Population, treatment, outcome, comparator and horizon are written down.
- 02
The method explicitly handles selection, contamination and simultaneous changes.
- 03
The interval of uncertainty is converted into an economic decision.
- 04
Extrapolation beyond the test is documented separately.
Limits
What this analysis does not prove.
- A badly executed experiment can be less credible than a good quasi-experiment.
- Rare or weak effects require samples sometimes incompatible with the calendar.
- The average effect can hide significant heterogeneity between segments or areas.
References
Works cited.
- Imbens, G. W. & Rubin, D. B. (2015), Causal Inference.
- Kohavi, R., Tang, D. & Xu, Y. (2020), Trustworthy Online Controlled Experiments.
- Gordon, B. R. et al. (2019), Marketing Science.