It’s well known that psychology can be “WEIRD.” We’re not talking about weird study results, like a recent one showing that chewing gum increases cognitive performance. Instead, we’re talking about WEIRD study methods, like the fact that scientific studies are often overwhelmingly White, Educated, Industrialized, Rich, and Democratic. WEIRD people represent 80 percent of behavioral-science study participants but only 12 percent of the world’s population. These samples are not only unrepresentative of humans in general, but in many cases they are outright outliers. However, because WEIRD researchers can easily select WEIRD samples–often their colleagues or students–the phenomenon persists.
With retail experimentation, we often see an analogous problem. Many retailers use skewed and inaccurate sample selections in their A/B tests, leading to skewed and inaccurate results. We call these samples DUMPY because they are often Driveable, Unusual, Metropolitan, Popular, and Young:
Using DUMPY samples in retail, though often convenient, can be catastrophic. DUMPY samples rarely represent a retailer’s fleet. And if a sample fails to represent the fleet, the test will be inaccurate. Relying on inaccurate results can tank revenue and reduce profit when the initiative is rolled out. For example, if you were to test the effect of new products on stores that are metropolitan, new, popular, and high traffic, you may see them performing well in these DUMPY samples. However, when you roll out these new products more broadly you could learn that they perform poorly in stores that are more rural, older, lower volume, and have lower-than-average customer satisfaction.
MarketDial is a tool that helps clients design tests perfectly and easily, every time. Our application uses advanced analytics to select test stores that are representative of your fleet. Avoid DUMPY samples, reach out for more information.