Get Access to our Extensive Library of A/B Testing Content and Resources
Implications of Multiple Control Matches on A/B Testing Accuracy and Frequency
We conducted a simulation experiment to examine the effect of multiple control matches on A/B testing accuracy and frequency. Our simulation experiment shows that 1-to-many matching offered slight improvements in noise reduction when compared with 1-to-1 matching. However, the improvements are not substantial enough to warrant a decrease in testing frequency, representativeness, and comparability that occurs in 1-to-many matching conditions. We recommend 1-to-1 matching as the best method to optimize return on investment for A/B retail testing. We conclude with recommendations for retailers and future researchers.