Validating an MMM: preferring a usable model to a seductive forecast.
Directly verifiable evidence
Primary sources to check the key claims.
- Meridian, model diagnostics (opens in a new tab)
Convergence, fit, validation and the limits of predictive metrics.
- Meridian, assessing model fit and results (opens in a new tab)
Separating predictive fit from causal validity.
Evidence status
What was not reproduced.
The source's training and out-of-sample evaluation were re-run, and return the values it publishes: training R² of 0.949, holdout R² of 0.843, holdout RMSE of 0.53 M, on the announced split of 96 learning weeks and 13 evaluation weeks. What is verified here are therefore its predictive performance measures — neither the causal structure it advances, which did not come out of the training, nor the three methods it compares itself with.
Key distinctions
Three conditions before calculating.
Out-of-sample prediction
Reserve a period or use a rolling time validation. A high in-sample fit can simply reproduce seasonality and trend.
Structural plausibility
Examine signs, delays, saturation, decomposition and contribution. An excellent MAPE does not authorise an inexplicable negative media effect or a curve that grows without limit.
Decision validation
Test whether the ranking of the scenarios stays stable when priors, windows, control variables and periods change reasonably.
Method
Organising validation as a series of gates
- 01Control the data
Check definitions, variation, breaks, joins, costs, calendars and coverage before interpreting the coefficients.
- 02Test the generalisation
Measure error, bias and interval coverage on weeks the model did not use.
- 03Falsify the mechanisms
Introduce a temporal placebo, remove periods, change the adstock windows and look for changes of sign or of rank.
- 04Calibrate and bound the use
Compare with the experiments and declare the authorised decisions, the channels that are too uncertain and the observed spend range.
Two models, one defensible validation
Model A predicts slightly better but fails on signs, stability and calibration. Model B is less spectacular and more usable.
| Diagnosis | Threshold | Model A | Model B |
|---|---|---|---|
| Out-of-sample MAPE | < 10% | 5.9% | 6.8% |
| 80% interval coverage | 70–90% | 58.3% | 91.7% |
| Plausible signs | 100% | 80% | 100% |
| Variation across specifications | < 25% | 48% | 14% |
| Gap to calibration | < 20% | 45% | 9% |
| Stable allocation ranking | > 75% | 54% | 83% |
Choosing 6.8% error rather than 5.9%
Model A obtains 5.9% MAPE, but attributes a negative effect to brand search, varies by 48% depending on the window and overestimates a geo test by 45%. Model B obtains 6.8% MAPE, plausible signs, a variation of 14% and a calibration gap of 9%.
The predictive gain of A is 0.9 point, but three decision diagnostics exceed the blocking thresholds. B covers 11 weeks out of 12 within its 80% interval and keeps the same allocation ranking in 83% of the variants.
Retain B for planning under constraints and reserve A for an investigation of specification, without using it to move the budget.
Acceptance conditions
What must be true to act.
- 01
An unused period measures error, bias and coverage.
- 02
Signs, delays and saturation are plausible or explicitly explained.
- 03
The results are stable across several reasonable specifications.
- 04
The experimental calibration and the range of use are published.
Limits
What this analysis does not prove.
- A good out-of-sample result does not guarantee the causal identification of the contributions.
- Channels with little variation or always active remain hard to separate.
- Calibration does not automatically correct a poor data structure.
References
Works cited.
- Jin, Y. et al. (2017), “Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects”.
- Chan, D. & Perry, M. (2017), “Challenges and Opportunities in Media Mix Modeling”.
- Gelman, A. et al. (2013), Bayesian Data Analysis.