Join CEO Morgan Davis as he makes the case for testing.
Register for the webinar.

Contact Us Client Portal
  • Blog Posts

Make mistakes or prevent mistakes?

By MarketDial / Jun 28, 2022

The best recipe for A/B testing
Lemons and oranges: A/B multivariate testing tools
Insight nuggets: A/B testing data science

The best recipe for A/B testing

A/B testing may sound complicated, but it’s as straightforward as evaluating if something is bitter or sweet. You take an idea, change the recipe a little, and then test it against the original recipe (the control group) to see which is preferred. That is A/B testing in a nutshell.

Perhaps the best example of why A/B testing is essential for any business is Coca-Cola’s epic 1985 recipe-change debacle. Amidst the cola wars of the 1980s, Coca-Cola knew something had to change. Its sales had been slipping for over a decade to PepsiCo, and blind taste tests revealed consumers preferred the sweeter taste of Pepsi to Coke. With this knowledge it seemed logical to assume that by making Coke sweeter, the product would be more desirable and competitive.

After 99 years of using the same formula, Coca-Cola decided to change the recipe, branding the replacement formula “New Coke.” But while the company had successfully A/B tested consumers’ flavor preferences, they had failed to A/B test consumers’ emotional preferences before rolling out the new product, assuming flavor alone would contribute to sales lift.

They were wrong. The resulting consumer outcry led to what had been deemed the “marketing blunder of the century.” As the company, itself, describes it, “The fabled secret formula for Coca-Cola was changed, adopting a formula preferred in taste tests of nearly 200,000 consumers. What these tests didn’t show, of course, was the bond consumers felt with their Coca-Cola — something they didn’t want anyone, including the Coca-Cola company, tampering with.”  

In the end, both consumers and Coca-Cola came to appreciate the value of the original product, and the company’s faux pas taught the marketing industry an important lesson: fully test ideas before rolling them out.

Lemons and oranges: A/B multivariate testing tools

Multivariate is another word that can seem more complicated that it needs to be. This simply means you test multiple variables at the same time. It is sometimes called multi-cell testing or A/B/C testing.

In 1747 James Lind conducted one of the first A/B multivariate tests. Believing that the cure to scurvy might be citrus fruit, he gave some sailors lemons and oranges but left the control group on the regular diet. In this example, the multivariate was testing both lemons and oranges. Lind could have run two separate tests where he tested only lemons with one group of sailors and only oranges with another group. This may have been more informative, proving conclusively that citrus fruit made the difference. But he chose to test both lemons and oranges at the same time.

While the results of his tests did indeed show marked improvement in the citrus group over the control group, Lind’s results were not immediately given credence. In fact, it took over 40 years for the British navy to distribute citrus to sailors.

When testing multiple variables at a time, it is therefore important to have the right A/B multivariate testing tools – tools that can help isolate the requisite variables, with qualified data supports that can help identify which variable combinations may add clarity and which may complicate results.

Lynn Zhang, Manager of Data Science at MarketDial notes that “it’s tempting to try to test pricing, merchandising, and marketing campaigns as a holistic effort – and then try to back out each impact individually. It’s just not reliably possible to generate trustworthy findings. Compare that unfocused effort with this clear one: Will a new visual design for our coffee endcap generate positive revenue lift for coffee products?”

Consider, for example, a restaurant wants to change multiple items on its menu. If it changes those items and sees a lift in sales, it will be difficult to differentiate which items, specifically, led to the lift; however, if the restaurant is testing the overall effect of the combined changes, then separating the individual variables become less important and as many variables as needed can be changed without compromising the integrity of the results.

Insight nuggets: A/B testing data science

Without testing, companies often lack the data needed to make fully-informed decisions. With testing, they gain nuggets of insights that can save them millions.

While James Lind may have conducted one of the first A/B tests, Claude C. Hopkins in 1927 is credited with utilizing A/B testing in advertising. He understood that acting on one’s instinct or impressions was insufficient for determining sales lift. In Scientific Advertising he writes, “One ad compared to another, one method with another. Headlines, settings, sizes, arguments, and pictures are compared. To reduce the cost of results even one per cent means much … no guesswork is permitted. One must know what is best.”

Morgan Davis, MarketDial CEO also emphasizes how A/B testing helps prevent poor decisions and emboldens great decisions: “It’s almost never one big, bad decision that sabotages a company. It’s often years of small decisions. Hypothesis testing can magnify these otherwise unforeseen outcomes. Without this knowledge retailers are hesitant to try the new, scary-but-effective things and continue to invest in the old, reliable-but-ineffective things.”

Consider some of the following use cases:

  • Product changes
    A candy company wants to add a jalapeño chocolate bar to its candy line but is unsure how consumers will react.
  • Merchandising
    A grocer is unsure if moving seasonal items to the front of the store will impact seasonal sales.
  • Customer loyalty
    A convenience store wonders if adding perks to its loyalty card will increase gasoline sales.
  • Promotions
    A retail store wonders if they should run “buy one get one free” on overstocked items or “buy one get one half off.”
  • Pricing
    A restaurant wonders if increasing the price of wine will impact dessert sales.
  • Employee incentive
    A fast-food chain wonders if annual bonuses will aid in employee retention.

When it comes to innovative ideas, the scope of possibilities are limitless. When it comes to productive ideas, however, definitive data is essential for guiding best practices.

For brick-and-mortar retailers, the stakes have never been higher, and there isn’t much breathing room for failed, costly initiatives. Knowing the expected sales lift – whether it be positive, neutral, or negative – can prevent erroneous decisions and promote successful decisions. Take a page out of Coca-Cola’s revised playbook, and know before you roll.

For more insights into how A/B testing works and how it can benefit your initiatives, consider these resources:

How to create a statistically valid test
How to see what is and what is possible
A bad test is worse than no test at all