It’s well known that psychology can be “WEIRD.” We’re not talking about weird study results, like a recent one showing that chewing gum increases cognitive performance. Instead, we’re talking about WEIRD study methods, like the fact that scientific studies are often overwhelmingly White, Educated, Industrialized, Rich, and Democratic. WEIRD people represent 80 percent of behavioral-science study participants but only 12 percent of the world’s population. These samples are not only unrepresentative of humans in general, but in many cases they are outright outliers. However, because WEIRD researchers can easily select WEIRD samples–often their colleagues or students–the phenomenon persists.
With retail experimentation, we often see an analogous problem. Many retailers use skewed and inaccurate sample selections in their A/B tests, leading to skewed and inaccurate results. We call these samples DUMPY because they are often Driveable, Unusual, Metropolitan, Popular, and Young:
- Drivable: Retailers often test in stores that are within driving distance of their headquarters. Though this may please executives who want to drop in on their way home from work, these stores often receive special attention, making them operationally unique.
- Unusual: Retailers often select stores with unusually high customer satisfaction. It may feel nice to highlight stores you know are thriving, but these stores may skew the results because, by definition, not all stores have unusually high customer satisfaction.
- Metropolitan: Retailers often select stores that skew metropolitan rather than rural. Though these stores are often more accessible, they are often unrepresentative.
- Popular: Retailers sometimes select test stores with high volume and traffic. Though these stores produce a lot of data, they may be outliers.
- Young: Retailers often select test stores that are new and flashy. These stores may be top-of-mind since they represent outsized investment, but the outsized investment may make them a poor choice for a sample group.
Using DUMPY samples in retail, though often convenient, can be catastrophic. DUMPY samples rarely represent a retailer’s fleet. And if a sample fails to represent the fleet, the test will be inaccurate. Relying on inaccurate results can tank revenue and reduce profit when the initiative is rolled out. For example, if you were to test the effect of new products on stores that are metropolitan, new, popular, and high traffic, you may see them performing well in these DUMPY samples. However, when you roll out these new products more broadly you could learn that they perform poorly in stores that are more rural, older, lower volume, and have lower-than-average customer satisfaction.
MarketDial is a tool that helps clients design tests perfectly and easily, every time. Our application uses advanced analytics to select test stores that are representative of your fleet. Avoid DUMPY samples, reach out for more information.
