Marketing experimentation: choosing a protocol that answers the decision.

SUPPORTED DECISIONChoose the simplest protocol that identifies the effect useful to the decision, then refuse any causal conclusion if the conditions of comparison are not verifiable.

Evidence status

What was not reproduced.

The source's power and sample-size calculations were re-run without substitution: 6 of the 8 values it publishes are reproduced to the digit, the other 2 within 1.2% and 1.9% — a gap of the order of the tolerance its own method declares, on a calculation that involves randomness. The campaign data of Liu, Bettaney and Chamberlain, their figures and their empirical results, by contrast, were never published: they cannot be re-run, by us or by anyone.

Key distinctions

Three conditions before calculating.

Start from the decision

The protocol depends on the change considered, on its unit of application and on the threshold that would alter the action. An available metric is not enough to define the experiment.

Protect the counterfactual

The comparison group must represent what would have happened without the intervention. Contamination, budget shifts and simultaneous changes weaken that comparison.

Pre-write the interpretation

Population, metric, horizon, exclusions and the rule to continue, stop or investigate further are written before the results are read.

Method

Choosing an experiment through four checks

01Define the unit of assignment
Identify the smallest level at which the action can be applied without major leakage — individual, account, store, area or period.
02Assess randomisation
If random assignment is feasible and stable, use a controlled test; otherwise document the assignment rule and the assumptions required for a quasi-experiment.
03Measure power and contamination
Set the minimum useful effect, the variability, the number of units, the duration and the risks of diffusion between groups. An underpowered experiment does not become conclusive with time alone.
04Connect the result to the action
Compare the observed interval with the decision threshold. An imprecise result calls for a new measurement or a prudent decision, not for artificial certainty.

DECISION TOOL

The protocol pre-selection matrix

Five situations make visible the candidate protocol, its horizon and the prerequisites that block any causal conclusion if they are missing.

Unit	Randomisation	Contamination	Candidate protocol	Fail-closed prerequisite	Horizon
Individual or account	Yes	Low	Randomised test	Assignable units and controlled spillover	Pre-test, test and follow-up
Area or store	Yes, by clusters	Medium	Geo test	Enough comparable areas and a stable pre-period	Pre-period, test and follow-up
Planned rollout	No	Measurable	Difference in differences	Untreated group and documented parallel trends	Sufficient pre-period and follow-up
Single market	No	High	Synthetic control	Credible donor pool and sufficient pre-treatment fit	Long pre-period and follow-up
No comparison	No	Unknown	Descriptive measurement	No defensible counterfactual	Observation only

If a prerequisite is missing, the output stays descriptive.

This matrix shortlists a protocol family. It replaces neither the power calculation, nor the diagnosis of assumptions, nor the review of the analysis plan.

CSVDownload the protocol selector↓

ILLUSTRATIVE EXAMPLE · SIMULATED DATA

A six-week geo test does not automatically prove a national extension

01 · SITUATION

Illustrative example: an activation is deployed in 12 test areas and compared with 12 control areas. Average weekly revenue before the test is €200k per group. The average weekly gap observed during the test is €12k, with a weekly interval from −€4k to €28k.

02 · CALCULATION

The central point represents 12,000 ÷ 200,000 = 6%. The interval crosses zero and includes effects that are very different for the decision. The central value is therefore not enough to authorise a generalisation.

03 · DECISION

The result stays undetermined. The decision is to extend or redesign the test if the minimum useful effect justifies the cost, rather than to extend the activation on the central estimate alone.

Acceptance conditions

What must be true to act.

01
The unit of assignment matches the lever genuinely controllable.
02
The comparison, the exclusions and the metric are defined before the result.
03
The risk of contamination and the simultaneous changes are documented.
04
The decision uses an interval and a business threshold, never the central point alone.

Limits

What this analysis does not prove.

A local experiment does not guarantee the same effect on another population or period.
Platforms can apply learning mechanisms that connect the groups.
A quasi-experiment depends on assumptions that are not entirely testable.
The absence of a detected difference does not prove the absence of any effect.

References