Geo test: matching markets before moving the budget.
Directly verifiable evidence
Primary sources to check the key claims.
- Google Research, Measuring Ad Effectiveness Using Geo Experiments (opens in a new tab)
Designing and interpreting geographic experiments.
- Google Research, Robust Causal Inference with Paired Geo Experiments (opens in a new tab)
Robust iROAS estimation and sensitivity to assumptions.
Key distinctions
Three conditions before calculating.
A unit without leakage
The geography must limit media exposure, travel, catchment areas and digital spillovers between treatment and control. An administrative border is not always a behavioural border.
A comparable pre-trend
The pairs must move together before the test on outcome, distribution, price and major events. A high average correlation is not enough if the gaps drift.
Decision power
The minimum detectable effect must be compared with the economic threshold. A test unable to distinguish +4% from zero cannot settle a decision that is profitable from +2%.
Method
Designing a geo test that can actually conclude
- 01Define the useful effect
Write the outcome, the horizon, the population and the minimum uplift that repays the campaign cost; that value becomes the power target.
- 02Build the pairs
Match markets on level, trend, seasonality, distribution, price and competitive mix, and reserve atypical periods for robustness tests.
- 03Assign and stabilise
Randomise the treatment within the pairs, freeze the schedule, control the other levers and monitor exposure leakage during the test.
- 04Analyse the difference
Estimate the treated-control gap, its interval, the pre-trends and the placebos, and report the cost per incremental unit and the influential markets as well.
Twelve pairs of comparable markets
The pairs are ranked by pre-test similarity. Power depends on the variance of differences within pairs, not only on the number of areas.
Twelve pairs detect an effect of about 3.9%
The historical standard deviation of within-pair differences is 4.8%. The plan keeps twelve pairs, a two-sided risk of 5% and a target power of 80%.
Approximate minimum detectable effect = (1.96 + 0.84) × 4.8% ÷ √12 = 3.88%. The minimum expected economic gain is 3.2%.
The test remains slightly underpowered for the economic threshold. Pairs must be added, the period extended, or the variance reduced through pre-test covariates.
Acceptance conditions
What must be true to act.
- 01
The economic threshold is defined before the power calculation.
- 02
The pairs have stable levels and pre-trends over several periods.
- 03
Assignment within the pairs is randomised and documented.
- 04
Leakage, distribution changes and local events are monitored.
Limits
What this analysis does not prove.
- A small number of markets makes the inference sensitive to one influential pair.
- The estimated effects may not transport to very large cities or to untested areas.
- National campaigns, media without geographic targeting and travel can contaminate the control.
References
Works cited.
- Vaver, J. & Koehler, J. (2011), “Measuring Ad Effectiveness Using Geo Experiments”.
- Kerman, J., Wang, P. & Vaver, J. (2017), “Estimating Ad Effectiveness Using Geo Experiments in a Time-Based Regression Framework”.
- Imbens, G. W. & Rubin, D. B. (2015), Causal Inference for Statistics, Social, and Biomedical Sciences.