Podcast

How to know which stores matter: uncover insights by drilling down

Having a measurement plan in place before you even start your marketing plan or campaign is cardinal rule of marketing and a best practice in analytics. However, sometimes it’s necessary to adjust that measurement plan, look past the aggregate data and gain key insights through microanalysis.

Jan-Eric and Zack discuss a recent example of how one client wasn’t seeing the impact from marketing on their business and instead of looking at the data across all stores, we drilled down to the store level, analyzing why some responded to marketing while others did not, and learned insights about demographics, competition, pricing structure and operations, that the CMO could focus on.

Listen here:


(Subscribe on iTunesStitcherGoogle PlayGoogle PodcastsPocket Casts or your favorite podcast service. You can also ask Alexa or Siri to “play the Uncovering Aha! podcast.”)

Welcome to Callahan’s Uncovering Aha! podcast. We talk about a range of topics for marketing decision-makers, with a special focus on how to uncover insights in data to drive brand strategy and inspire creativity. Featuring Zack Pike and Jan-Eric Anderson.

Jan-Eric
Hi, I’m Jan-Eric Anderson, chief strategy officer at Callahan.

Zack:
And I’m Zack Pike, head of data at Callahan.

Jan-Eric:
Welcome back to the podcast. Today we’re going to talk about a lesson that Zack and I have recently learned and it’s really, it was an eye opener for us because it really challenged something that we’ve advocated in the past and it’s a good reminder of sometimes it’s good to forget what you think you know and be willing to change your mind on things.

Jan-Eric:
The topic around this is something you’ve probably heard or come across in your professional life, which is, when you’re going to do a marketing campaign or marketing plan and you’re heading into, one of the cardinal rules is to make sure that you have a measurement plan in place before you start so that you can determine, you know what your KPIs are, how you’re going to measure success, what your measurement plan is. If you have got all that figured out before you started your campaign, you’re ahead of the game. So it’s an idea of being prepared and having a plan ahead of the time so that you can be objective as you go into evaluate success. Seems like a safe rule of thumb, a good cardinal rule.

Zack:
It is a best practice in analytics, but it’s everywhere.

Jan-Eric:
No question about it. It’s hard to argue against that. But that’s actually what we’re here to do is argue against that today. We have had a recent experience working with a client where we ran into a situation where what we thought was the best way to analyze and approach looking at some marketing tests and marketing effectiveness and media effectiveness actually was not telling us the true story. And I think that the headline of all of this is really that when we went into it, we were doing some media tests. We were running media in certain parts of the country and expecting to see some sort of impact on the business. And we kept getting the question from the client, what’s the impact of my media? And we kept running into a dead end. The reality is, we weren’t seeing any impact on the business and we were having a really hard time accepting that.

Zack:
Right. Yeah. Yeah. It’s been, we’ve always had this saying that we use for the past two years, which is no two stores are created equal. So we always are trying to break things apart in analysis as much as we can. But even in this scenario, it wasn’t deep enough. And we’re finding that to be true in a lot of areas. So, it might make sense for me just to talk about the analysis and kind of break apart how we did it and then we can …

Jan-Eric:
Yeah, I think so. And, you know, the analysis we were doing ahead of time and the traditional kind of wisdom on how to look at something like this, the client has multiple points of distribution and is a retail concept and has multiple points of store locations. What we were trying to see is, and our approach into this was looking at sales in the DMAs and in the time periods when media was running right. And that’s not rocket science, that’s pretty a widely accepted way of looking at it. Yet every time we looked at it, we would see really no difference in those DMAs or during those times, those windows of time when media was on versus control areas or control DMAs or control weeks. We were not seeing any separation.

Jan-Eric:
So, to your point that no two locations are created equal, they all kind of have a mind of their own, that sort of started getting us thinking, maybe we’re looking at this in aggregate too much. Maybe we need to be drilling down, if we have the data at the individual location level, why wouldn’t we dive a little deeper?

Zack:
Yeah. And you know, location level, DMA level analysis is something lots of people are doing. But what was happening in this case is the wash was our problems. So you had stores that were doing well and stores that were doing poorly. And when they washed out, the net impact on the business was essentially non-existent. It was kind of inconclusive. While that is kind of an insight in itself, this was a point where we took a step back and said, okay, there’s variance in how stores perform, but is there something that could tell us if a store benefits in the first place? Do they even realize benefit from all this media and marketing stuff we’re doing or are they just a store that’s gonna perform the way they’re going to perform and the stimulus in the market doesn’t really impact what’s going on for that store?

Zack:
And there’s a lot of ways you could go about an analysis like this. You could start to define hypotheses around that. Well, I think that stores that have a certain square footage are all going to perform the same way or stores that have a certain type of owner are all going to perform the same way. Or you could take an approach and letting math determine it for you. So that’s the approach that we took. We ran an analysis against all of the stores. And the point of the analysis was not, in the beginning, not to determine the ROI or how much benefit was produced in a store by the campaign. It was to determine if there was benefit at all. Because it gets back to this question of I know half my marketing’s working and half doesn’t. I just don’t know which half.

Jan-Eric:
And when I hear that, I always think about this, that maybe TV works and radio doesn’t work or maybe online video works but social doesn’t work. And it’s truly trying to figure out which channel is better.

Zack:
Yes. And in this scenario, what if all the channels were working, but some of your areas where you were actually generating sales just weren’t working.

Jan-Eric:
Completely changes your perspective on the whole, I know part of it’s working, part’s not. It’s less about channel, it’s more about two specific locations.

Zack:
Exactly. It’s where does the customer actually acquire my product? Is that working or is that not working? And then we layer in the stimulus on top. So we went through that work, hundreds of stores, and we were able to tease out that there were, depending on how you look at the data, there were anywhere from 50 to a 35% of stores that realized zero benefit from any of the media that we had run over the past roughly two years. And so that was really insightful. Now you knew, there was a lot of work that went into it, but now you knew, here’s my stores that when I’m evaluating media, I should probably be looking at these stores closer than my stores that I see no benefit from. 

Zack:
And then, that’s one aspect to this on how you use this going forward, but the other big pieces, now I can start to tease out why these stores are different, right? Which is where the real benefit of this comes because I might be able to tease out something I can control. We want every store to benefit. That’s how our dollars are working the hardest. So at that point you can start looking at all of your attributes of the store. If it’s a retail environment, I may be looking at the location, I may be looking at the level of competition in their area, I may be looking at the type of store owner, how seasoned they are in the business, how much experience they have in this type of industry.

Jan-Eric:
Concentration of populations, the demographic profile of those people that live around the store. And this is where it’s nice to have a good diverse set of data that is available to be aggregated or split apart at that, kind of at that geography level, at that base store level.

Zack:
Yup. So once you start to tease out those differences, and in this case, we found some really interesting stuff. We found some things that the company can actually control, which is where it gets valuable. Through that analysis, you can start to then say, okay, well now I know that these let’s say hundred stores benefit from media, these 40 stores do not. So, I’m going to start asking questions around how I’m allocating budget. Should I be allocating as much money to these non-benefiting stores as I have been? Should I be allocating any, right?

Jan-Eric:
Where do I apply my attention? So as a company, as a CMO, I start to understand the profile of a store that will benefit most from my efforts in marketing. I would think as CMO, your hope is to have the greatest amount of impact on the business. So the more stores and locations that we can get to fit that profile, we put them in a position to benefit from the marketing efforts. That’s the greatest influence and impact a CMO can have.

Jan-Eric:
So this type of analysis is critical for actually trying to get exponential growth on the impact that marketing can have on the business. Right? You look at those, the characteristics, and I know from this example, we were seeing, as we were building a profile of the benefiting store location, we were seeing characteristics that had to do with things they can’t control, that the client can’t control, including things like the demographic makeup and the presence of competition around that store location in that trade area. We were also seeing things that the client can control. Things like pricing structure. Things like operations. And so, there were components there as well, and those are things that a CMO could focus on.

Jan-Eric:
So, building that up really kind of puts us in a position to then be able to say, is this a location, if we find a location then that’s not benefiting, can we make the necessary adjustments here to get it into that group, and should we focus our effort there?

Zack:
Right.

Jan-Eric:
Or should we give up on them? And that’s a really controversial conversation to be having because obviously, you don’t want to give up on any of your locations of your business, but it’s better to know where you can and cannot have impact, where is there potential to have impact. And you fold that into conversations around how are we going to get to x percent positive growth next year. Well it could be in fixing the following locations that had been contributing zero to growth in the past.

Zack:
And it also empowers the CMO to know what stores they can actually impact in relation to that question. So if we’re trying to grow the business a certain percentage next year, typically, it’s hey, from marketing, I need this much. Well now, I have the data to say how much I actually can produce because I know where I’m producing the money. And through this analysis, we also knew how much was being produced. And it actually changed some of the ways that the client started looking at the channels and the value of their channels because now we had a better lens into what this benefit actually was. And some of the channels that historically had been thought of as the ones producing all the revenue turned out not to actually be the ones producing all the revenue.

Jan-Eric:
Yeah. So you and the team were able to tease out essentially through this analysis how many widgets were being sold as a result of different channels. And were able to kind of create an order of magnitude but also to understand kind of how much was being contributed on a channel by channel basis, which is fascinating because you’re able to do that at the store level, at each location level essentially.

Zack:
Right. Right.

Jan-Eric:
And then, if I’m not mistaken, also then layered in media cost information to start to understand an ROI on that lift knowing that it’s within those stores.

Zack:
Yup. Because now you have a much better understanding of waste. I know where my waste is really clearly in an analysis like this. And ROI gets clearer because I’m looking at my return on investment on the stores where I know I’m seeing benefit. And then you put those two together and you get the total. But yeah, it’s changing I think very quickly the way that we look at most of our clients. We’ve done this on a handful of clients now and so far it’s proving out to be really interesting stuff and it’s making us a lot smarter for planning.

Jan-Eric:
One of the things that I found fascinating in that, in that analysis is knowing the details of it was that there was one particular media channel that has been kind of the favorite or assumed biggest contributor to business. And there was no question about it. It had the power to sell a lot of product and move a lot of product when it was on. But the reality is the cost of it and the waste factor of it due to the limited number of locations that were actually getting benefit from it, it made it so upside down from an efficiency standpoint that it’s hard to justify even continuing to do it moving forward. Fascinating analysis.

Jan-Eric:
So again, I think that the, why this is such a relevant topic is again, this really calls into question this best practice of just knowing out of the gates, this is how we’re going to measure this. Had we just stuck by our guns and said, nope, what we’re going to do is we’re going to pull the DMAs, we’re going to pull the time period and we’re going to look at sales comparison of our test versus our control. And if the test group isn’t up, then it didn’t work. We would be absent of finding this treasure chest of insight of understanding what’s really working and what’s not working.

Zack:
And I think that’s the whole thing about this little conversation we’re having here is you have to have a certain level of faith in whoever you’re working with on the data side that they’re going to look at things the right way when it comes time to look at it. Now, I’m not saying there should not be a plan for measurement. There should always be a plan. You should always, you need to have your data set up and all the stuff that we’ve talked about in several of these podcasts. But you have to have a level of trust to say there’s lots of ways you could peel back the layers of this onion and lots of different orders you could do it.

Zack:
And on the analyst side, you also have to be flexible enough to be able to think about things in different ways. It would have been easy for us to do what we had always done. It was very difficult for us to take this different approach, that when we started it, we didn’t even know if it was going to produce anything. You go down these paths and it’s like, okay, well, let’s try this. It may not work but let’s try it. I think that gets back to some of the conversations we had about relationship between the agency and the CMO and all that stuff. It all has to be there for something like this to work.

Jan-Eric:
Well, and you’ll be humble about this, but your team turned this around very, very quickly and would not have been able to do that where it not for having tech in place that allows this type of analysis to happen quickly, but also the tech is no good if you don’t have data available in there that’s been aggregated and is ready to be analyzed in unforeseen ways. So you had it broken down to structured, which puts you in a position to be able to do an analysis like this. We kind of needed a time to figure out that this was the right thing to do.

Zack:
Yeah.

Jan-Eric:
So, thanks for joining us on the podcast. Hopefully this conversation, as we’ve talked about a key lesson that we’ve just learned, really opened our eyes to challenge a best practice in the marketing analytics world that we hadn’t really thought about before. But hopefully you find this insight helpful and thanks for joining.

 

You’ve been listening to the Uncovering Aha Podcast. Callahan provides data savvy strategy and inspired creativity for national consumer brands. Visit us at callahan.agency to learn more.