Sarah Jacobsen, Data Scientist, MarketDial
Tests don’t always turn out the way we hoped but still provide us with the information needed to fail fast and move forward. On average, over 60% of in-store tests aren’t clearly successful, even at the most mature testing organizations. It’s normal! If all tests are significantly positive, that becomes suspect. Negative or neutral tests help us keep us honest and let us know that the process is producing accurate results.
Testing is not always about driving revenue, it’s also about saving the company from making costly mistakes. If you receive negative or neutral results from an in-store test, it’s important to approach the situation with an open mind and use the results as an opportunity to learn and improve. Here are some effective ways to respond to undesirable results.
Try not to take it personally
Ultimately, retailers are trying to find initiatives that are in the best interest of the customer. Focusing on the customer instead of the idea can help to depersonalize unsatisfying outcomes. Begin by asking, “What was the intention or goal of the test?” This question can produce significant insights. Some possible responses might include:
- To learn more about the initiative in order to replicate or expand it in the future. If this was your goal, that’s great! MarketDial can help you find what stores, areas, or dates the initiative was successful in and iterate from there.
- To know what the impact would be before rolling it out to all sites. Utilize the impact analyzer to estimate the impact to all sites. Did the test have negative results in every location or just some locations? Does a partial rollout make sense?
- To appease a business leader’s request / check off a to-do box. Educate yourself on the benefits of testing and then help your business leader understand what decisions and actions make the most sense based on test outcomes.
- To test multiple phases. If testing is happening in phases, remember that the results of any one test are just one piece of the puzzle. Keep looking at each phase and resist making a final assessment until all phases are complete.
- To see how severe or drastic negative outcomes might be. Many times there is an awareness that an initiative will not produce positive outcomes, and testing is more a matter of assessing how significant those negative outcomes will be. In this scenario, negative or neutral results prove a hypothesis correct and lend insight into the full scope of the impact.
- To verify if a vendor’s prediction on sales increases is accurate. Any outcome provides the retailer with information requisite to have meaningful discussions and negotiations with vendors.
Analyze the results
Start by analyzing the results of the test. Look at the data to understand the differences in performance between the treatment and control sites. Consider factors like engagement rates, conversion rates, and user behavior. Take a deep dive into the following metrics:
- Custom charts and lift explorers. Examine what products are driving different lifts and which group of sites are driving lift.
- Lift by site. What sites produced negative lift and which produced positive lift?
- Lift over time. Were the results positive at any point in the test?
- Overall store (all products). Consider the state of the stores and products at the onset of the test and throughout the test period. Were they struggling to begin with or underperforming in general throughout the test period?
- Overall category. Go one step higher in the hierarchy. Is that category struggling?
Identify the problem
Once you’ve analyzed the results, identify the specific problem areas, whether it be one area, one store type, one time period, or likely a variety of factors. Try to understand why one variation performed better than the other, and what factors may have contributed to the differences in performance. Fail fast means, in part, that you’re able to see what went wrong and move forward quickly.
Consider making changes and re-testing the new iteration
Based on your analysis, make changes to the underperforming variation. This might involve making changes to the messaging, the design, or the user experience. After you’ve made the changes, test again to see if your improvements have had a positive impact. Continue to iterate and improve until you achieve the results you’re looking for.
Be willing to scrap the initiative all together
Sometimes being able to let go of an idea is the most cost-effective way to proceed. Investing too much time or energy into something that is unlikely to produce dividends can be counterproductive to growth.
Use the learnings to spawn new ideas
Insights breed insights. Brainstorm. Allow your understanding of the results to drive creative new approaches to the problems at hand.
Communicate with stakeholders
Throughout the process, keep stakeholders informed about the results of the test and the steps you’re taking to address any issues. This shows that you take the feedback seriously and are committed to delivering the best possible experience for your customers.
Stay focused on growth
Remember store testing is an ongoing process of iteration and improvement. Negative or neutral results are not a failure, but rather an opportunity to learn and grow. By approaching the results with a growth mindset and a commitment to improvement, you can fail fast, turning a negative test into a positive experience and achieving better results over time.
Learn more about the best in-store testing methods and solutions, read these:
Solving the omnichannel data conundrum
Key drivers for developing into a mature testing organization
Minimize bias and maximize your testing results
