Price test: size an experiment that can reach a decision.
Evidence status
What was not reproduced.
The synthetic educational calculation is verified by the specification, the fixture and Python. The methodological source Kohavi et al. is archived and its excerpts are verified, but its results, its randomisation, its interference mechanisms and its implementations are not reproduced.
Key distinctions
Three conditions before calculating.
Economic threshold
A new price may lose conversion while increasing contribution. The minimum acceptable rate is current contribution per visitor divided by the candidate margin.
Assignment unit
Assign by customer, account, area or store according to the risk of seeing several prices. Session-level assignment exposes the test to comparisons and group switching.
Power before duration
Duration follows from the number of units required, eligible traffic and the decision cycle. Stopping when a result looks favourable increases false positives.
Method
Designing a price test from margin to decision
- 01Calculate the threshold rate
Measure current contribution per visitor and candidate unit margin. Their ratio gives the minimum conversion that preserves short-term value.
- 02Choose the useful effect
Set the smallest difference that would genuinely change the decision, then define alpha and beta risks and any multiple-test correction.
- 03Size and assign
Calculate the sample per group, predefine exclusions before the test and keep price, promotion, assortment and communication stable during the experiment.
- 04Decide on contribution
Report conversion, margin, contribution per visitor, mix, cancellation and an interval. Non-significance is not proof of equivalence.
The detectable effect – sample size curve
The smaller the conversion gap to detect, the larger the sample required. The target is tied to the contribution threshold, not chosen for convenience.
Approximately 7,360 visitors per group to detect 1.5 points
Baseline conversion is 12%. The test targets a 1.5-point difference, two-sided 5% risk and 80% power. Margin rises from €30 to €36.
Approximation: n = 2 × (1.96 + 0.84)² × 0.12 × 0.88 ÷ 0.015² = 7,359 per group. Economic threshold conversion is 12% × 30 ÷ 36 = 10%.
The test can detect a 1.5-point decline, below the maximum decline of 2 points. It requires about 14,720 eligible visitors excluding predefined exclusions.
Acceptance conditions
What must be true to act.
- 01
Calculate the economic conversion threshold before testing.
- 02
The assignment unit prevents a customer from seeing multiple prices.
- 03
Size, duration, exclusions and stopping rule are preregistered.
- 04
The decision uses contribution and its interval, not conversion alone.
Limits
What this analysis does not prove.
- A short test measures renewal, reputation and repeat-purchase effects poorly.
- Customers may share prices across groups and change behaviour.
- The approximation must be replaced by simulation for complex metrics.
References
Works cited.
- Kohavi, R., Tang, D., Xu, Y. et al. (2020), “Online randomized controlled experiments at scale”, Trials 21, 150 (CC BY 4.0) — full text. (opens in a new tab)
- Kohavi, R., Tang, D. & Xu, Y. (2020), Trustworthy Online Controlled Experiments.
- Imbens, G. W. & Rubin, D. B. (2015), Causal Inference for Statistics, Social, and Biomedical Sciences.
- Deng, A., Lu, J. & Chen, S. (2016), “Continuous Monitoring of A/B Tests without Pain”, KDD.