How to create a statistically valid test - Read The White Paper

Contact Us Client Portal
  • Blog Posts

Impact analytics: How risky is change?

By MarketDial / Jun 08, 2022

Impact data analytics: Seeing the back of your head
Impact model data analytics
The Arnold and Bohner model
The Isson model

Impact data analytics: Seeing the back of your head

In the movie Harry Potter and the Prisoner of Azkaban, Harry and Hermione travel back in time to save Buckbeak. While Hermione is normally hyper-focused on the task at hand, she catches a temporary glimpse of her head from behind and protests, “Is that really what my hair looks like from the back?” In that moment Hermione articulates what we rarely discuss: we only see the back of our heads in two dimensions – a metaphor that has relevance well outside the beauty industry.

Many businesses fail to see the back of their heads in a three-dimensional context. While they perform risk assessments to attempt to identify blind spots, they often negate a second, more important piece: impact analytics, or understanding exactly how particular risks or changes could impact the business as a whole. Before 2020, for example, the thought of a pandemic disrupting commerce was nothing more than a movie trope. It likely showed up on the radar of numerous risk assessments – a potentiality – but now in 2022, the impacts of that potentiality-turned-reality pandemic are still being felt.

Benjamin Franklin once noted, “Experience keeps a dear school, yet fools will learn in no other.” As brick-and-mortar retailers continue into an unpredictable future, reliable impact analytics models are essential for both mitigating negative impacts and maximizing opportunities. Retailers who use more than past experiences to inform future decisions – who understand the impacts of change in real time – are the retailers that will move from mediocre to remarkable in the wake of the unprecedented changes currently sweeping the retail industry.

Impact model data analytics

The Arnold and Bohner model

When it comes to assessing change impacts, two models stand out. The first was developed for software maintenance by Robert S. Arnold and Shawn A. Bohner. It has three core components: traceability, dependency, and experiential.

  • Traceability is primarily concerned with relationships, tracing requirements, specifications, and tests. Another name for this would be driver analyzer – or analyzing what drives the results of a test. For example, if a retailer tests a promotion in a low-income area and sees sales lift, but runs the same promotion in a high-income area and sees no sales lift, the income disparity is likely what is driving the lift disparity. Tracing these relationships is a key component to understanding the impact on sales in every area across your fleet.
  • Dependency focuses on the depth of the impact; does the impact reach beyond the key components of an assessment or test? A basket analyzer, for example, helps measure dependency. Basket analyzers consider what other elements in a group are impacted by change. If milk goes on sale, do egg sales also increase? If the store layout is changed to enable greater flow through produce, how is the bakery affected? Dependency shows the grand-scale impacts of changes on products that are often dependent or interconnected to one another.
  • Experiential pays attention to past experiences and expert insight. It considers back data. One convenience store, for example, utilized pre-pandemic behavior to craft a test that isolated post-quarantine sales lift from app-usage sales lift.

Therefore, according to the Arnold and Bohner impact model, strong impact data analytics solutions have the ability to identify both stand-alone and interrelated impacts. This model is particularly effective at pinpointing relationships between products or elements, and identifying how changes to each element influences overarching impacts.

The Isson model

The second effective impact model for data analytics was created by Jean-Paul Isson. Whereas the Arnold and Bohner model is focused on analytics that inform the impact of change, the Isson model is focused on how to take the analytics and make them actionable. It has six key components using the acronym IMPACT:

  • I: Identify the question. Often the quality of the question influences the quality of the answers. Lynn Zhang, Lead Data Scientist at MarketDial, compares generating a strong hypothesis to a road trip, “Before you start, you should know why you are going there, the route you will take, and what you’ll do once you’ve arrived.” Having clear, concise questions provides more definitive, reliable data. Isson also emphasizes that once a quality question has been identified, at this stage it is helpful to “set a clear expectation of the time and the work involved to get an answer.”
  • M: Master the data. High-quality solutions often integrate the aforementioned time frames into their algorithms. These time frames fluctuate based on the level of confidence in the results. For example, to reach 90% confidence, a test may need to run four weeks longer than if the confidence levels were 85%. Those conducting the tests can assess what levels of confidence they want relative to the time investment. Setting up strong test parameters—such as representativeness to avoid biases and equitable control groups also can ensure accurate data results. With strong testing platforms, such as MarketDial, these parameters are easily configured with the software doing most of the calculations automatically.
  • P: Provide the meaning. Similar to the Arnold and Bohner model, this step utilizes driver and basket analyzer tools to fully comprehend the data results. By looking at the interrelatedness of products, the data becomes more informative. If one product’s change provides lift in one area but decreases it in another, then the result is net neutral. Looking only at a singular outcome, could alter the meaning. A big-picture approach gives full context and informs the wisest decisions.
  • A: Actionable recommendations. The strongest impact retail analytics understand how crucial this piece is to the process. So, you have data showing sales lift–now what? Do you roll out the change across your entire fleet or just in specific locations? What had the greatest impact on sales lift and is that impact geocentric in any way? Once you fully understand the data, then you can make informed decisions about the best way forward for implementing the change.
  • C: Communicate insights. While infographics are always insightful, even the best testing platforms out there can benefit from person-to-person communication. That’s because behind every number is a person. An increase in sales happened because people chose to purchase more of that product. Data contains information about people. But numbers are not the people themselves, and interpersonal communication gets to the heart of the matter more than computational communication. Having a strong client services team that can discuss outcomes and communicate insights with you is therefore essential to maintaining a human-centric approach to healthy change implementation and impact.  
  • T: Track outcomes. When change is prominent, consistent research and evaluation is necessary. What may be avant-garde one year often becomes passe the next. Or, one hypothesis, once proven, could create more hypotheses, leading to evolutionary business growth. Continuing to test can help monitor all the impacts of change, answer progressive questions, and guide well-informed decisions.

While we cannot easily see the back of our heads in three dimensions, we can easily see all facets of brick-and-mortar businesses from multiple dimensions. No matter what impact model you prefer, by utilizing effective testing tools such as MarketDial, impact data analytics can become second nature, lending robust, dimensional insights into change impact – providing actionable insights that inform best practices in the wake of change.

If you would like to learn more about how testing can impact data analytics, check out these articles:
Testing to thrive in today’s retail landscape
Actionable insights: Taking test-and-learn to the next level
When people take precedence: the data science behind retail decisions