Price test: size an experiment that can reach a decision.
Answer:
A price test must do more than detect a conversion difference. It must distinguish a change large enough to alter contribution. Price simultaneously changes margin, selection, basket, cancellation, mix and sometimes long-term perception. The experiment plan therefore starts from the economic threshold, calculates the required sample and fixes a decision rule before results are known.
Decision:
Run and interpret a price test only if its power covers the economically acceptable conversion loss and exposure does not contaminate the groups.
Principles:
A new price may lose conversion while increasing contribution. The minimum acceptable rate is current contribution per visitor divided by candidate margin. Assign by customer, account, area or store according to the risk of seeing multiple prices; session-level assignment exposes comparisons and group switching. Duration follows the required number of units, eligible traffic and the decision cycle. Stopping when a result looks favourable increases false positives.
Method:
Calculate current contribution per visitor and candidate unit margin; their ratio gives the minimum conversion preserving short-term value. Set the smallest effect that would genuinely change the decision, then define alpha and beta risks and any multiple-test correction. Calculate sample size per group, predefine exclusions and keep price, promotion, assortment and communication stable. Report conversion, margin, contribution per visitor, mix, cancellation and an interval: non-significance is not proof of equivalence.
Worked example — approximately 7,360 visitors per group to detect 1.5 points.
Baseline conversion is 12%. The test targets a 1.5-point difference, two-sided 5% risk and 80% power. Margin rises from €30 to €36. Approximation: n = 2 × (1.96 + 0.84)² × 0.12 × 0.88 ÷ 0.015² = 7,359 per group. Economic threshold conversion is 12% × 30 ÷ 36 = 10%. The test can detect a 1.5-point decline, below the maximum decline of 2 points. It requires about 14,720 eligible visitors excluding predefined exclusions.
Decision rules:
Calculate the economic conversion threshold before testing. The assignment unit prevents a customer from seeing multiple prices. Size, duration, exclusions and stopping rule are preregistered. The decision uses contribution and its interval, not conversion alone.
Limits:
A short test measures renewal, reputation and repeat-purchase effects poorly. Customers may share prices across groups and change behaviour. The approximation must be replaced by simulation for complex metrics.