MMM calibration: bringing the model closer to a relevant experiment.
Directly verifiable evidence
Primary sources to check the key claims.
- Meridian, model configuration and calibration (opens in a new tab)
Priors, calibration periods and model specification.
Key distinctions
Three conditions before calculating.
Estimand compatibility
A test often measures a local, marginal variation while the MMM estimates a historical average return. Spend, period, population, outcome and transformation must be matched.
Preserved uncertainty
The test carries sampling error, contamination and transport. Calibration must weight its precision and add the extrapolation uncertainty, not impose a constant.
Validation outside calibration
An experiment used to calibrate cannot on its own serve to validate the model. Keep separate periods, areas or tests to check generalisation.
Method
Calibrating an MMM coefficient without erasing the differences
- 01Write both estimands
Document treatment, counterfactual, population, outcome, horizon and scale for the MMM and for the experiment; refuse the calibration if the gap cannot be corrected.
- 02Transform the scale
Convert the experimental effect into the model's variable and transformation — incremental sales, iROAS, elasticity or response coefficient.
- 03Combine the precisions
Use a likelihood, an informative prior or a penalty whose strength reflects experimental variance and transportability; publish the uncalibrated model.
- 04Test the propagation
Check prediction, residuals, contributions, response curves and allocation. A good local calibration must not degrade other channels or periods.
Before, experiment, after: the calibrated interval
The wide observational estimate is drawn closer to a more precise test. The posterior result keeps an uncertainty instead of copying the experimental figure.
The test tightens the estimate without replacing the model: posterior standard deviation 0.23.
JSONDownload the calibration calculation↓The iROAS moves from 2.40 to 1.44 without being forced to 1.30
The uncalibrated MMM estimates 2.40 with a standard deviation of 0.65. A compatible experiment estimates 1.30 with 0.25, transport included. Both estimates are treated as normal and independent.
Precisions: 1 ÷ 0.65² = 2.37 and 1 ÷ 0.25² = 16. Combined mean = (2.40 × 2.37 + 1.30 × 16) ÷ 18.37 = 1.44. Posterior standard deviation = √(1 ÷ 18.37) = 0.23.
The test dominates without completely erasing the historical information. The allocation is recomputed on 1.44 with an approximate interval from 0.98 to 1.90.
Acceptance conditions
What must be true to act.
- 01
The estimands of the test and of the MMM are compared line by line.
- 02
The transport uncertainty is added to the experimental error.
- 03
The uncalibrated model stays published as a point of comparison.
- 04
Prediction, contributions and allocation are revalidated after calibration.
Limits
What this analysis does not prove.
- A local experiment can poorly represent national saturation.
- Tests selected because they are favourable create a calibration bias.
- Independence between the experiment and the MMM data can be imperfect.
References
Works cited.
- Chan, D. & Perry, M. (2017), “Challenges and Opportunities in Media Mix Modeling”.
- Vaver, J. & Koehler, J. (2011), “Measuring Ad Effectiveness Using Geo Experiments”.
- Sun, Y. et al. (2017), “Geo-Level Bayesian Hierarchical Media Mix Modeling”.