A Bayesian MMM makes the assumptions visible and the decisions probabilistic.
Directly verifiable evidence
Primary sources to check the key claims.
- Meridian, introduction to priors (opens in a new tab)
Role of priors, regularisation and Bayesian updating.
Evidence status
What was not reproduced.
The paper's model was re-trained and reproduces the performance measures it publishes. No Bayesian inference was run for all that: the paper's model is not Bayesian, and the Bayesian method it compares itself with was not re-run. The prior and posterior calculations presented on this page are therefore not reproduced.
Key distinctions
Three conditions before calculating.
Traceable prior
Every prior states source, population, period, scale and reason for transport. Internal experiments take precedence over generic benchmarks; physical and economic bounds regularise without inventing excessive precision.
Identifiability before computation
A high-performing sampler does not separate two perfectly correlated channels. Variation, seasonality, support, collinearity and definition changes are diagnosed before choosing the family of priors.
Posterior decision
The mean of a ROAS is not enough. Compute probability of profitability, risk of loss, distribution of marginal curves and regret of the allocations. The rule depends on risk aversion and reversibility.
Method
Building a governable Bayesian MMM
- 01Specify the generating process
Define outcome, controls, trend, season, adstock, saturation, interactions and hierarchical level; tie each component to a testable business assumption.
- 02Elicit the priors
Convert experiments, studies, comparable benchmarks and constraints into distributions; publish plausible quantiles rather than arbitrary precision, and provide a weak reference prior.
- 03Fit and diagnose
Check convergence, effective sample size, autocorrelation, divergences and posterior prediction; test recovery on simulated data and the ability to reproduce held-out periods.
- 04Calibrate and decide
Compare the posteriors with the experiments, run sensitivity analyses on the priors and simulate the decisions; authorise an allocation only if it stays acceptable across the credible variants.
From prior knowledge to posterior decision
Three channels combine prior and data; the final rule bears on a probability of profitability, not on a point coefficient.
The prior avoids a false ROAS of 4.1 on a collinear channel
Weekly data give video a highly unstable ROAS of 4.1 ± 3.0, because its spend follows the season. A comparable earlier experiment estimates 1.8 ± 0.5. The prior retained is 1.9 ± 0.8, in order to stay wider than the experiment.
With a normal approximation, prior precision 1/0.8² = 1.5625 and data precision 1/3² = 0.1111. Posterior mean: (1.9 × 1.5625 + 4.1 × 0.1111) ÷ 1.6736 = 2.05. Posterior standard deviation: √(1/1.6736) = 0.77. The probability of a ROAS above the threshold of 1.6 is about 72%.
The channel is not extended to the maximum: 72% stays below the gate of 80%. A geo experiment is funded to increase the information before the next allocation.
Acceptance conditions
What must be true to act.
- 01
Sources, scales and transport of the priors are documented.
- 02
Convergence, prediction and parameter recovery are verified.
- 03
Decisions use threshold probabilities and risk of loss.
- 04
Several credible priors and an experimental calibration are compared.
Limits
What this analysis does not prove.
- A precise prior can dominate weakly informative data.
- Numerical convergence guarantees neither causality nor correct specification.
- The posteriors remain conditional on the variables, transformations and data chosen.
References
Works cited.
- Gelman, A. et al. (2013), Bayesian Data Analysis, 3rd ed.
- Jin, Y. et al. (2017), “Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects”, Google Research.
- Vehtari, A., Gelman, A. & Gabry, J. (2017), “Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation”, Statistics and Computing.