Can retail take test and learn to the next level?
Software solutions for the brick-and-mortar market
Test and learn, then what?
The three pillars of the prove-and-improve framework
Moving beyond test and learn to prove and improve
Morgan Davis, MarketDial CEO
In 1675 when most scientists were using a compound lens microscope, Antony van Leeuwenhoeck was grinding and polishing bi-convex lenses and blown-glass lenses, reaching a magnification level of 275x. In looking at the world through many new lenses, he made a unique discovery which donned him the father of microbiology. Naming his new finding “animalcules,” Leeuwenhoeck explained, “I discover’d living creatures in Rain water, which had stood but few days in a new earthen pot, glased blew within.”
Three centuries later in the late 1980s, the CEO of Capital One, Rich Fairbank, decided it was time to see business through a new lens. He believed every decision – from marketing to product design to customer satisfaction and selling decisions – should be held under the microscope of scientific experimentation, a belief that donned him the father of the agile model known as Test and Learn.
Testing initially was not the easiest of endeavors for companies. But this merging of science and business dovetailed with the entrance of the internet and e-commerce onto the global stage. Online retailers quickly grasped onto online data analytics as a sure-fire test-and-learn methodology for directly addressing the problem of tracking and predicting consumer behavior. Analysts had immediate answers to their questions: “What buttons and what path did users take to get to our product? Where did they go afterwards, and what other things did they look at?” Data could be isolated very cleanly, making it easy to identify which change had which outcome.
Software solutions for the brick-and-mortar retail market
The test-and-learn options for brick-and-mortar retailers has remained a challenge. Online data access undoubtedly has been serving the e-commerce industry well, but it has not been meeting the needs of the offline industry. Because changes in the offline space are often bigger and more complex than in the online space, a test-and-learn approach for offline retailers has historically been more complicated. Online analysts can make small changes and see real-time outcomes. They can change a button or a page layout and easily test and learn how users respond to that change. But offline retailers may want to change the entire physical layout of their site to optimize the space for customers. These are big, costly changes.
SaaS companies realized this gap in the market in the late nineties and created early software to try to help brick-and-mortar retailers generate viable data for their diverse needs, but it was clunky with complex programming, needing data scientists well-trained in how to use it. While much of that software has undergone updates to modernize it, it’s still clunky and complex in ways that fail to meet offline retail needs. But cutting-edge, emergent software systems, such as MarketDial, are now utilizing machine learning to handle much of the heavy lifting of test-and-learn complexity for data scientists and analysts, enabling brick-and-mortar retailers to now have similar data access as that of their online counterparts.
Never was this software more essential. What was ground-breaking in the eighties, has become mandatory: the science of business. Companies no longer see the scientific method as separate from or unique to high-caliber business practice. They understand that if they are not testing regularly then they most certainly are investing in bad opportunities and missing out on good opportunities. That said, many younger companies still struggle to know when to begin testing.
Businesses don’t get started without art and luck. But the day after they get started, companies need to turn into science mode. Too many brands think they’re still in the “we’re too small phase” when they’re not, and they wait too long to start investing in reliable data.
The worst-case scenario is when retailers make unwise, uninformed gut decisions, but the impact is not immediately obvious. It’s almost never one big, bad decision that sabotages a company. It’s often years of small decisions. Hypothesis testing can magnify these otherwise unforeseen outcomes. Without this knowledge, retailers often are hesitant to try the new, scary-but-effective things and continue to invest in the old, reliable-but-ineffective things.
Test and learn, then what?
In the history of retail, we are living in unprecedented times. Not only is the industry still navigating the balance between online sales, brick-and-mortar sales, shifting consumer preferences, and direct brand-to-consumer purchasing options, it is also facing challenges impacted by the COVID-19 pandemic, from staffing to supply chain disruption and inflation. Because of these unique challenges, many are starting to feel the test-and-learn model is no longer sufficient. Retailers are finding they need more than informative insights gleaned from testing; they need actionable insights gleaned from a proven hypothesis. The time has come to shift our thinking from a more static approach of test and learn to a more hands-on approach of prove and improve.
In the same way Leeuwenhoeck used a new lens to gain insight into what was otherwise not visible, retail needs a new framework to clearly identify the underlying risks and opportunities facing them. The traditional test-and-learn framework has focused on the idea that says, “We have a theory. We’re going to test it, and we’re going to learn from that test.” The prove and improve framework goes deeper into the science of data to develop a targeted hypothesis, subject that hypothesis to A/B testing with a well-defined control, minimize noise and bias, analyze the results, and take appropriate action based on that knowledge. This framework doesn’t just want to test a hypothesis and learn from the results of that test; it wants to prove a hypothesis correct or incorrect and then use actionable insights to improve processes based on that newfound knowledge.
Proving or disproving a hypothesis is essential to improving overall retail initiatives. Hypotheses can seem logical on the surface but fail to represent reality. A retailer may think that eliminating a product could cut back on costs, for example, but miss the bigger picture that cutting that specific product deters customers from choosing that store over a competitor. Good data can help retailers have courage to try what would otherwise seem risky and avoid risks that might otherwise seem practical. Even neutral results can be informative. Knowing that a decision won’t affect consumer behavior either way can still inform the final solution.
The three pillars of the prove and improve framework
The prove and improve framework has the ability to transform brick-and-mortar retailers, giving them access to actionable insights. This framework relies on three primary pillars: the testing needs to be consistent, repeatable, and generalizable.
- Consistent means everybody within a retail organization should be running experiments for every potential change, and the tests should all be run with the same methodology. Systemic consistency leads to systemic growth and improvement.
- Repeatable means that no matter how many times an experiment is run, the results remain the same. Reliable testing methods ensure repeatable outcomes.
- Generalizable means that the results of the experiment can be universally applied and produce the same outcome. If it works on a smaller scale, retailers should be able to roll it out and have it work on a bigger scale. Retailers need to have confidence that they can go from test to rollout and have the results be the same.
For data to move from providing informative insights to actionable insights, it needs context. Knowing what the temperature is outside, for example, is just information. But understanding how the weather will impact things such as commutes gives it context, creating actionable insights. By creating studies that start with a strong hypothesis and are consistent, repeatable, and generalizable, the data outcomes are given the context they need to become actionable.
Before Leeuwenhoeck discovered his animalcules, he first had the courage to look at the dregs of blown glass differently. Value can be found anywhere if we know how and where to look for it. Just like Leeuwenhoeck magnified rainwater and discovered a formerly unseen world of microbes, by implementing a prove-and-improve model, we now have the ability to magnify retail dilemmas and bring processes into tight focus, enabling us to discover new ways to propel our businesses forward with actionable insights.
Want more information? Try these resources:
Test today when others won’t, so tomorrow you can succeed where others can’t
Testing to thrive in today’s retail landscape
