Marketing Mix Modeling: specify the decision before estimating the model.
Directly verifiable evidence
Primary sources to check the key claims.
- Official Google Meridian documentation (opens in a new tab)
MMM workflow, data, modelling, analysis and optimisation.
Evidence status
What was not reproduced.
The source's dataset and generator were checked, and the paper's model was trained in full. It reproduces the three performance values it publishes — training R² of 0.949, holdout R² of 0.843, holdout RMSE of 0.53 M — on the same data and the same split (190 regions, 109 weeks, 13 channels, of which 13 weeks are held back for evaluation). The causal structure the paper puts forward does not come out of it: the graph obtained distinguishes no relationship between channels. The three methods the paper compares itself with were not re-run.
Key distinctions
Three conditions before calculating.
Economic estimand
The target must correspond to the value to be optimised: net sales, margin, new customers or contribution. The model states whether the effects are incremental, average or marginal, and how media costs are treated.
Identifiability
Variation, collinearity, support, number of observations and confounders determine what the model can separate. A plausible coefficient under a strong prior is not information entirely produced by the data.
Decision validation
Predictive error is not enough. Sign, order of magnitude, stability, decomposition, marginal slope and the performance of a retrospective allocation must be checked, then calibrated with compatible experiments.
Method
Writing the model contract and enforcing it
- 01Frame the decision
Define budget, channels, units, periods, scenarios, constraints and update frequency; write down the forbidden decisions, notably extrapolations outside the support or a creative reading that is too fine.
- 02Audit the signal
Test variation, correlations, breaks, missing data, price, promotions, distribution, seasonality and events; build an identifiability matrix before any estimation.
- 03Estimate and challenge
Compare several specifications, priors and windows; examine residuals, contributions, adstock, saturation and parameters; refuse economically impossible trade-offs even if prediction improves.
- 04Calibrate and govern
Match the estimates with compatible experiments or quasi-experiments; publish intervals, domain of use, version log, gaps to the evidence and the validation decision by level of exposure.
The decision contract of an MMM
Eight fields connect decision, data, model, validation and guardrails; each field has an acceptance gate.
Budget, levers, horizon
clearedMarginal contribution
clearedVideo–search correlation
conditionalPrediction + decision
clearedCompatible geo test
cleared−15% / +20% observed
blockingQuarterly review
clearedBounded pilot
selectedA predictive model with 7% error fails at allocation
An MMM on 156 weeks predicts sales with 7% MAPE. It recommends moving €400k from video to search. Video and brand search have a correlation of 0.82; the video budget has never been below €250k per quarter.
Under specification A, video ROAS is 0.9 and search 3.4. Under a longer adstock prior and a geographic calibration, video is 1.8 ± 0.7 and search 1.9 ± 0.5. The recommendation of −€400k places video at €120k, outside the support. A scenario limited to −€120k produces €96k of expected gain with an interval from −€18k to +€205k; the maximum regret is €130k.
MAPE does not validate the initial allocation. The contract forbids the cut outside the support and turns the move into a €120k test. The decision is "pilot with a guardrail", not "optimise automatically".
Acceptance conditions
What must be true to act.
- 01
The contract names the decision, the estimand and the domain of use.
- 02
Identifiability is assessed before the coefficients are interpreted.
- 03
Predictive validation and decision validation are separated.
- 04
Any recommendation outside the support becomes a bounded test.
Limits
What this analysis does not prove.
- Aggregate data separate synchronised activities with difficulty.
- Priors can stabilise without creating new information.
- Structural changes reduce the transportability of historical curves.
References
Works cited.
- Jin, Y. et al. (2017), “Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects”, Google Research.
- Chan, D. & Perry, M. (2017), “Challenges and Opportunities in Media Mix Modeling”, Google Research.
- Brodersen, K. H. et al. (2015), “Inferring Causal Impact Using Bayesian Structural Time-Series Models”, Annals of Applied Statistics, 9(1), 247–274.