Deciding with little data: buying the information that can genuinely change the action.
Evidence status
What was reproduced, and what it does not prove.
The CSV and the Python run verify the net values of information and the ranking in the illustrative scenario. They validate neither the probabilities nor the performance of a method on a real market.
- Technical specification of the learning portfolio with sparse data, specification in French (opens in a new tab)
v1, 31 August 2026 · CC BY 4.0 · specification in French · Method document written by marketing-mix.net. The figures are illustrative: they are not market measurements.
Key distinctions
Three conditions before calculating.
Decision before data
Collection starts from the action to be chosen, the exposure and the date. Data useful for explaining the past may change nothing in the imminent choice.
Light triangulation
With few observations, combining unit economics, a targeted test, customer research and an external benchmark often produces a more robust decision than a single model.
Value of information
A measurement is valuable if it has a real chance of altering the action and avoiding a material loss. Precision without decision consequence has little value.
Method
Building a profitable learning sequence
- 01State the tipping points
Write the assumptions that would change the go/no-go, the budget or the target, and attach to each a range and a loss if it is false.
- 02Create the evidence options
List pilot, holdout, geo test, survey, operational observation and model; cost each one with its delay, population and expected result.
- 03Compute the net value
Estimate the probability that the evidence changes the decision multiplied by the avoidable loss, then subtract the cost and the value of the delay, and rank the options.
- 04Learn sequentially
Buy the first piece of evidence, update the assumptions and recompute the value of the following ones; stop when information can no longer change the action.
The cost × value-of-information portfolio
Four evidence options are compared on cost, lead time, probability of changing the decision and avoidable economic loss.
The geo test is worth €45k net; the MMM is premature
A launch exposes €300k. A geo test costs €45k and has a 30% chance of avoiding the full loss. A customer holdout costs €25k and has a 25% chance of avoiding €150k. An MMM costs €90k, but the twelve months available do not separate the channels.
Net value of the geo test: 30% × €300,000 − €45,000 = €45,000. Holdout: 25% × €150,000 − €25,000 = €12,500. The MMM receives a tipping probability close to zero as long as the variation is missing, hence a net value close to −€90,000.
The geo test is run first, then the holdout is reassessed with its result. The MMM is deferred until a history and identifiable variations are available.
Acceptance conditions
What must be true to act.
- 01
Every collection is tied to an explicit decision tipping point.
- 02
Cost, delay, tipping probability and avoidable loss are quantified.
- 03
Evidence is bought sequentially, not all at once.
- 04
A method that cannot be identified is deferred rather than forced.
Limits
What this analysis does not prove.
- The probability of changing the decision is itself uncertain.
- The value of information falls if the commercial window is short.
- Several small correlated pieces of evidence are not worth several independent ones.
References
Works cited.
- Vaver, J. & Koehler, J. (2012), “Periodic Measurement of Advertising Effectiveness Using Multiple-Test-Period Geo Experiments”, Google Research.
- Chen, A., Longfils, M. & Remy, N. (2021), “Trimmed Match Design for Randomized Paired Geo Experiments”, Google Research.
- Owen, A. & Launay, T. (2016), “Multibrand Geographic Experiments”, Google Research.