Choosing a marketing method by estimand, bias and decision.
Key distinctions
Three conditions before calculating.
Object estimated
An attributed ROAS, an experimental lift and an MMM contribution do not designate the same quantity. Numerator, denominator, window, population and counterfactual are written down before results are compared.
Identification before precision
A narrow interval around a biased estimator is not strong evidence. Randomisation, discontinuity, exogenous variation or control assumptions define what is identifiable; sample size then determines precision.
Compatible decision
A method can be excellent at detecting an anomaly and inadequate for allocating an annual budget. Frequency, granularity and the domain of transport must match the rhythm and the perimeter of the decision.
Method
From the business question to the evidence design
- 01Write the estimand
Specify treatment, outcome, unit, population, comparator and horizon — for example, the average incremental effect of a national video campaign on weekly net sales over eight weeks among eligible households.
- 02Map the biases
List selection, targeting, seasonality, anticipation, interference, incomplete measurement and compositional change, then assess which ones the design removes, reduces or leaves intact.
- 03Compare feasibility and resolution
Assess cost, lead time, power, granularity and repeat frequency; an underpowered experiment can be less useful than a robust quasi-experiment, and an MMM can illuminate the portfolio but not a specific creative.
- 04Plan the triangulation
Pair a primary method with a calibration source, and define in advance how a divergence will be handled — revisiting perimeter, assumptions, transport or timing, without averaging automatically.
The estimand–method–bias map
Six methods are compared on the counterfactual, the grain, the horizon, the dominant bias and the decision they can genuinely support.
| Method | Object estimated | Counterfactual | Decision | Identification |
|---|---|---|---|---|
| Attribution | Allocation | Rule | Execution | low |
| A/B test | Average effect | Randomisation | Product / price | strong |
| Geo test | Aggregate effect | Control areas | Media | strong |
| Quasi-exp. | Local effect | Exogenous variation | Policy | medium |
| MMM | Contribution | Conditional model | Portfolio | medium |
| Survey | Self-report | Response | Diagnosis | low |
Three ROAS answer three different questions
A search campaign shows an attributed ROAS of 6.2. A pause test on comparable areas estimates an incremental ROAS of 1.8 ± 0.6. The annual MMM attributes a ROAS of 2.4 ± 0.9 to the channel, including part of the delayed effects.
Attribution shares out €620k of observed sales for €100k of spend. The test estimates €180k of sales that would not have happened, with an interval of €120k to €240k. The MMM estimates €240k over a longer horizon, but partly confounds brand search and video activity. At a 35% margin, the central incremental contribution of the test is €63k, that is €0.63 per euro spent.
The 6.2 ROAS remains useful for steering queries and traffic quality; it does not justify the budget increment. The test supports the immediate marginal decision and the MMM informs the annual balance. The divergence becomes a synergy hypothesis to be tested.
Acceptance conditions
What must be true to act.
- 01
The estimand is formulated before the method is chosen.
- 02
The dominant bias and the counterfactual are published with the result.
- 03
The decision stays within the observed domain or justifies its transport.
- 04
A divergence between methods triggers a diagnosis, never an automatic average.
Limits
What this analysis does not prove.
- No map replaces examining the actual design and data.
- The best theoretical method can be unexecutable or insufficiently powered.
- Effects rarely transport without assumptions across segments, periods and intensities.
References
Works cited.
- Hernán, M. A. & Robins, J. M. (2020), Causal Inference: What If.
- Imbens, G. W. & Rubin, D. B. (2015), Causal Inference for Statistics, Social, and Biomedical Sciences.
- Pearl, J. & Mackenzie, D. (2018), The Book of Why.