Podcast

Attribution modeling is bullsh*t

Callahan Agency | June 4, 2018

Attribution modeling is a very misunderstood subject. It’s a very misunderstood activity.

Beware the downsides:

  • With attribution modeling, the expectation of what you’ll get back is not actually what’s delivered.
  • Attribution modeling not take into account outside variables that are effecting the business.
  • Attribution modeling does not move at the speed of business and does not turn around insights as quickly as needed.

Still planning to fall back on attribution modeling? Don’t answer until you’ve listened to this podcast.

Listen here (or subscribe on iTunes,  StitcherGoogle PlayGoogle PodcastsPocket Casts or your favorite podcast service):

Welcome to Callahan’s Uncovering Aha! podcast. We talk about a range of topics for marketing decision-makers, with a special focus on how to uncover insights in data to drive brand strategy and inspire creativity. Featuring Jan-Eric Anderson and Zack Pike.

Jan-Eric:
Hi! I’m Jan-Eric Anderson, Chief Strategy Officer at Callahan.

Zack:
I’m Zack Pike, Vice President of Data Strategy and Marketing Analytics at Callahan.

Jan-Eric:
Today’s episode is going to be of particular interest to any listener who is either currently working with an attribution model, or is considering investing in an attribution model. Zack and I have both worked with clients and vendors who do attribution models. Zack, I know you’ve got some thoughts on this. What do you think about attribution models?

Zack:
I think it’s all bullshit.

Jan-Eric:
Wow. So you’re not a big fan is what you’re saying, I guess. That’s maybe another way of saying that. Well, okay so maybe we’ve lost some listeners at this point already.

Zack:
Yeah.

Jan-Eric:
But that’s okay. We gotta get this off our chest, so let’s talk a little bit about that. We’re not a big fan of attribution models; maybe it’s because there’s some things to be aware of. So let’s start by, maybe it’d be good to qualify that statement by defining what you actually mean by an attribution model.

Zack:
Yeah. I think I said to you, “Are we ready to get hate mail on one of our podcasts?”

Jan-Eric:
I’m ready, I already get it.

Zack:
So yeah, so there are some uses for attribution modeling. The problem is that it’s a very misunderstood subject. It’s a very misunderstood activity, but yeah let me define what we’re talking about. So there are different ways to figure out attribution in a business. When we talk about attribution, we’re trying to figure out the key levers, or the activities that are contribution to performance of the business, whether that’s revenue or traffic or whatever.

And what we’re talking about here specifically is marketing channel attribution modeling. That is trying to assign a revenue value to every channel that was executed in a marketing campaign, over a number of years, or over a campaign or some flight window or whatever the parameters of the analysis are. We’re not talking about a financial attribution analysis, where the finance team is understanding the value of the pricing increase on top of the product mix, on top of an issue in the product that happened last year, where you’re really taking a full look at the entire business. Where marketers sometimes get into trouble with attribution modeling is in channel attribution modeling, which is what we’re really focused on.

Jan-Eric:
Yeah, and so to have a very small window of view there, when there are other things that can be impacting the business basically, that’s where we can get into trouble.

Zack:
Yeah, and I think, you know the biggest thing that I think about first when a client starts asking about attribution modeling or you know I’ve worked on teams that have done these models in the past, and had to communicate these results to clients is that is back to the misunderstanding pieces. Sometimes these things run under a false flag of what they actually do. And so the gold standard when we think about attribution is okay, I know who Zack is, and I know all of the activities that Zack has taken that involve me, from a marketing standpoint. I know the –

Jan-Eric:
As a consumer.

Zack:
As a consumer, I know the media that first introduced Zack to the brand, I know the paid search link that he clicked that eventually lead him to the website. I know the store that he went in to and why he went there to purchase the product. I know why he returned the product, I know why he then repurchased and on down the years of my engagements with Zack — I can see all of that activity. That is attribution modeling.

If we had that, things would be very different. The problem is we don’t have that. It’s that the technology required to connect all of those pieces still in the majority of business situations just isn’t there because there’re too many gaps in the measurement mechanism to get there.

In something like an e-commerce only business, it’s a bit easier. We’ve got the digital footprint of that whole journey in some cases, but in most businesses, especially if you don’t control your distribution channel, if you don’t collect all your customer information at that point of sale, attribution is really, really difficult and –

Jan-Eric:
It is difficult to do, and so the attribution may end up really being a generalization of assumptions being made and spit out represented in terms of being this is what really this channel is driving. And there’s danger in that, because that can end up having profound impact on decisions being made about how to allocate dollars and these decisions are being made on gross assumptions that might be loose connections at best.

Zack:
Yeah. Those assumptions are critical, and at least in my experience, in most of these models that I’ve seen, we’re leaving out too many variables in that equation to really, where someone like me would, feel confident in the results that paid search is worth $4 million to the company over the last year. That’s where we’re going down to.

Jan-Eric:
Yeah.

Zack:
I’d be interested though, in what your thoughts are, like you know what do you think about it?

Jan-Eric:
Well, you know, so many times, especially my background in directing clients or consulting clients on what to do with media, these attribution models end up, start taking the roles, the ultimate judgment of a media plan. And so the hierarchy of evaluation on media planning will be media post by, why did you get what you paid for, and were there high engagements or high clicks and video completion rates or whatever channel we’re talking about. That’s kinda the number one thing. But then the ultimate thing that comes through, and this is what puts media buyers and media planners on edge is when an attribution model company has been engaged to come in and build a report on what happens. ‘Cause now you’re sitting around on pins and needles, you don’t know what the report’s gonna be. It’s gonna come in way after the fact, and they’re going to give the ultimate number is that ROI number. How much revenue was driven compared to the amount of money invested in this channel?

As a planner, as a media planner you hate that, to be at the … subject to that evaluation. When you have no idea what the math is that goes into it, you can call into question the validity of the information in general. But then, the other thing, and this is I mean put the ego aside, I get really concerned because when that’s the only thing that it’s available to evaluate the effectiveness of the campaign, this starts to just automatically steer the media plan for the following year. And that’s fine as long as you have a lot of confidence that this is the right thing, but as you said when we opened, there are so many variables that are impacting the business that to take this single analysis and then say, “Hey, social media looked like it had the best ROI. Let’s dump of all our money into social next year”.

Zack:
Right.

Jan-Eric:
That gives me great, great, great concern. I guess that tease up kind of if running under the false flag is the number one kind of beef with an attribution model, the second one for me, is probably that it’s dangerous, that it runs … If it’s being evaluated in isolation of everything, of other things, there’s so many other things that can be impacting the business what can we attribute to business results; there’s so much more beyond the channel. So when channel attribution models are the lead or only thing being used, there’s a lot of danger in isolation.

Zack:
Yeah. There are a lot of things we can do from a measurement in evaluation perspective before ever talking about an attribution model.

Jan-Eric:
Right.

Zack:
Let me say this, we are not saying that companies who do attribution modeling are bad, or they’re lying to their clients, the data just isn’t there to do it the right way. And so, a good attribution analyst typically, when they present the model, will also package that with a set of testing recommendations. And those testing recommendations are there to help prove or disprove the hypothesis that are driven from the model. So if social is worth x amount of money to the company from a return standpoint, the analyst will probably propose “Hey, how about we test social in this market? Here is my hypothesis on what we should test. And then I’m gonna take those results to help validate what I’m telling you now.” And that’s the right way to do it.

However, we’re all smart people. If that’s the first time you’re doing testing, or talking about testing around social or media or paid search or whatever, then we’ve missed the boat a while ago. We didn’t need the attribution model to figure out good testing recommendations. It should almost be backwards, we should be testing for a while, and then doing attribution modeling because those test results are gonna help inform the model down the road.

Jan-Eric:
Yeah, and then you know, just connecting or identifying more of those variables that are impacting business.

Zack:
Yeah.

Jan-Eric:
Is it whether, have we had a price increase and there are varying levels of sensitivity to price increases? How [00:10:30] has income tax legislation maybe impacted business? That’s something that’s impacting our consumer, or our category. All these different types of varieties, competitive activity, etc., etc. There’s just so much more beyond just the mix of allocation of dollars between different channels.

Zack:
Yeah, you think about any company. We’re probably all familiar with YETI, right? They make the expensive coolers, and the drink [00:11:00] canisters and things. That is a company that is very dependent on their marketing strategy. They make some of the most expensive coolers in the world, and there’s a lot of people who say they’re just the same as the Coleman. And so their message is critical. But think about all the stuff that has to go right for YETI to be successful. They don’t control their distribution channel. Those things are sold in Cabela’s, and wherever else — Walmart and Home Depot and stuff. So the product has to be good, the message has to be there, the product has to be available to customers. They have to set their price right. There’re so many factors in that equation. They have to deal with Coleman saying that their coolers are just as good, at 20 bucks a cooler instead of 400. So then when you’re working on limited customer attribution data in the first place, trying to go down the paid search actually accounted for this amount of money, it’s bit of a stretch. It’s hard to be confident in that, without good testing.

Jan-Eric:
Yeah. Well, so testing and perpetual kind of ongoing learning is another interesting topic, and it leads me at least to my third main issue with attribution models. And that is the speed at which they move. In my experience, the timing of attribution models, the report when it comes back, when the company will show up to give the results of the model, or whatever the case may be. I feel like 9 times out of 10, it is just too late to impact what we’ve been working on. It’s never in sync, it seems with our planning cycles. So I guess to summarize that, my thought on that is it that one of my third beef for me with attribution models is that it just doesn’t move at the speed of business. And I need a long period of time of data to come in, to be able to have a substantial amount of data to go into the model, for the model to work. And then spit out results, at this point it’s too late. I needed the information as we were going, as we were adjusting, as we were moving through.

Zack:
And then you’re trying to use those results for the next year’s planning, right? ‘Cause you’ve already missed this year. And there’s arguments on the other side too. It’s all well if you’re planning once a year, you’re probably not catching all your benefits anyways, why are you planning once a year. Reality is that in many businesses there’s a big seasonal aspect to what’s happening. YETI coolers, I guarantee their business is seasonal. And you can think of any number of brands, there’s a level of seasonality to it, and that takes planning to make sure you hit the season at the right time, with the right channels and the right investment.

And so there’s arguments on both sides, but you’re right. So I get them at the end of the year, I’ve already done my planning, then I’m trying to gather up what I can to try and use any results for the following year, which is then even further out and potentially things have changed so substantially by then.

Jan-Eric:
Well, making small bets, fail fast, fail small fail fast. I had a former boss, he used to say, “I wanna do a little to learn a lot.” That mindset doesn’t fit very well in the package of an attribution model. You build a substantial large plan and approach, and then you apply attribution on a channel by channel basis. It seems very limiting, it feels very old school, and in this case I mean old school in a bad way.

Zack:
Sure.

Jan-Eric:
It feels like it’s out of date, and not in lock step with some other approaches that we would take with looking at marketing data and business analysis, and impacting businesses, we put different stimulus in the market place. And measuring it back, directly back to business impact when we put a social medial activation plan in Houston, Texas. How did the business respond in Houston, Texas, when that was the only variable that changed in that market?

Zack:
It’s critical. That is a critical point. And this is where attribution modeling can be valuable. A good attribution modeling company sometimes won’t even go into an analysis unless they can control the things that you’re talking about. So clearly defined tests, with control groups that we can compare against, and drive real math statistics based analysis. Unfortunately that doesn’t happen most of the time. Most of the time, what happens is a company is so excited about selling an attribution model, and the brand is so excited about buying it we jump into it, and the first step is “Okay, give us the last three years of data. Let’s gather up everything we can around media and marketing and sales and distribution, all this stuff. Let’s gather the last three years of data.” Where the modeling company had nothing to do with the measurement functions that were in place over the past three years.

If you took the approach of, “Okay, we are starting this engagement. Let’s help you from a measurement perspective over the next three years. Let’s put the structure in place to be able to measure things the right way, to do our test and controls the way that they need to be done.” The second aspect there is light years ahead of just going back and trying to figure out what we can figure out about the past, the history.

Jan-Eric:
So we’ve pretty much beaten the crap out of channel attribution modeling. Maybe it wasn’t the intention completely. Anything redeeming, to think about? You made the comment earlier, it’s not like they’re bad people doing this, or bad companies doing that. There’s a place for it.

Zack:
Absolutely.

Jan-Eric:
I would say that a very redeeming quality of an attribution model is in the absence of other information, it gives you something to lean on, to understand how a channel is performing, and gives you an idea, if nothing else, directionally how one channel may be contributing compared to another.

Zack:
Yeah. When we think about the totem pole of things that we should be doing from measurement, all of it is in the vain of improving our confidence in the decisions that we’re making. Even the best measurement function, the best evaluation of a campaign it’s still just improving confidence it’s not always answering the question for the points we’ve talked about earlier with so many factors impacting a business.

Jan-Eric:
You know, just kind of in summary we talked about some watch outs that a lot of times channel attribution modeling can run under a false flag. And maybe be the expectation of the buyer who’s investing in this, the expectation what they’re gonna get back is not at all actually what’s gonna be able to be delivered. Another watch out was that when channel attributions are used in isolation of other variables that are obviously impacting the business, there’s danger in looking at any analysis that is void of, or ignores, doesn’t take them into account some of those other variables. And the third main thing was that [00:18:30] it just doesn’t move at the speed of business. And turn around insights the way that we need it. And those are some of our watch outs. Zack, I don’t know about you, but I feel better for getting some of this off my chest. Maybe that’s an unintended benefit that came from this.

I hope you found this beneficial from a listening stand point, some things to watch out for.

I’m Jan-Eric Anderson, he’s Zack Pike and we’ll talk to you again soon.

 

Thanks for listening to our Uncovering Aha Podcast. Callahan provides data savvy strategy, and inspired creativity for national consumer brands. Visit us at Callahan.Agency to learn more.