Simulated marketing benchmark: learning to build a comparable interval.
Key distinctions
Three conditions before calculating.
A common definition
Numerators, denominators, windows and exclusions must coincide. An acquisition-only CAC is not comparable with a CAC that includes brand, sales and onboarding; a revenue ROAS is not a contribution.
A comparable population
Proximity is judged on unit economics, purchase cycle, maturity, channel, geography and intensity. An explicit similarity score is better than an over-broad sector label.
Distribution and updating
Median, quartiles and dispersion show the heterogeneity. The benchmark becomes a prior whose weight decreases as internal evidence grows, which avoids both blind imitation and the rejection of all external information.
Method
Building a benchmark usable for deciding
- 01Audit the metric
Write formula, unit, horizon, currency, date, source and exclusions; discard observations that cannot be reconciled, or keep them in a separate stratum.
- 02Segment the distribution
Filter on the variables that structurally change the metric, and publish sample size, median, interquartile range and extreme values — do not hide a small sample behind decimals.
- 03Weight the proximity
Give each observation a comparability weight based on margin, channel, maturity and market, calculate a weighted mean and test its sensitivity to excluding the dominant references.
- 04Update locally
Combine the external prior with the internal estimate according to their precisions; if the gap is large, look for a difference of definition, mechanism or selection before concluding underperformance.
From raw benchmark to comparable prior
Eleven simulated profiles illustrate filtering, weighting and updating. No figure in the file describes a real market.
Simulated data: a simulated ROAS prior is updated from 2.19 to 1.90
Eleven fictional profiles are built to illustrate a comparable distribution. Their simulated weighted mean is 2.19. The population weighted standard deviation — the sum of the weights multiplied by the squared deviations, divided by the sum of the weights — is 0.49. An equally fictional internal measurement is 1.70 with a standard deviation of 0.40.
External precision = 1 ÷ 0.49² = 4.16; internal precision = 1 ÷ 0.40² = 6.25. Combined estimate = (2.19 × 4.16 + 1.70 × 6.25) ÷ 10.41 = 1.90. Combined standard deviation = √(1 ÷ 10.41) = 0.31. Approximate 95% interval: 1.29 to 2.50.
The internal evidence, more precise in this example, moves the prior from 2.19 towards 1.90. The decision must use the internal contribution interval, not turn simulated profiles into a performance target.
Acceptance conditions
What must be true to act.
- 01
The formula and the window are identical between references and the internal case.
- 02
The distribution, the sample size and the filters are visible.
- 03
The weight of a benchmark decreases as internal evidence becomes more precise.
- 04
No target is set without translating the metric into local economics.
Limits
What this analysis does not prove.
- The eleven profiles are simulated and do not constitute a sector benchmark.
- Published data often suffers from selection of the best results.
- A similarity weighting remains a judgement and must be tested.
- A changing market can quickly make even a real distribution obsolete.
References
Works cited.
- Gelman, A. et al. (2013), Bayesian Data Analysis, 3rd ed.
- Rothwell, P. M. (2005), “External Validity of Randomised Controlled Trials”, The Lancet, 365(9453), 82–93.
- Pauwels, K. & Reibstein, D. J. (2016), “Challenges in Measuring Return on Marketing Investment”, Foundations and Trends in Marketing, 10(1), 1–107.