Validate an MMM: prefer a usable model to an attractive forecast.
Answer:
A Marketing Mix Model is not valid because it reproduces historical sales well. It must predict unused periods correctly, produce plausible effects, remain stable under reasonable perturbations, align with available experiments and support its intended decision without excessive extrapolation.
Method and decision:
First check definitions, variation, breaks, joins, costs, calendars and coverage. Measure error, bias and interval coverage on weeks the model did not use. Falsify mechanisms with temporal placebos, omitted periods and changed adstock windows, looking for sign or rank changes. Compare estimates with experiments and declare authorised decisions, overly uncertain channels and observed spend range. Accept, restrict or reject the model through these five diagnostic families, with blocking thresholds before allocation.
Worked example:
Model A has 5.9% MAPE but assigns negative effect to branded search, changes by 48% with the window and overestimates a geo-test by 45%. Model B has 6.8% MAPE, plausible signs, 14% variation and 9% calibration gap. A gains 0.9 predictive point but fails three blocking decision diagnostics. B covers 11 of 12 weeks in its 80% interval and keeps the same allocation ranking in 83% of variants. Retain B for constrained planning and reserve A for specification investigation, never for moving budget.
Limits:
A good out-of-sample result does not guarantee causal identification of contributions. Channels with little variation or always-on activity remain hard to separate. Calibration evidence itself has scope and uncertainty limits.