One of my great struggles in the entire data world is to get people to understand the difference in rate and value. It seems like this problem has a thousand different faces, yet it can be extremely difficult to find the right way to correct the misconceptions of any particular case. People are constantly trying to abuse data to show that they provided value or that something is directly tied to an outcome, despite the fact that the data itself can not in any way tell you this fact.
I was faced recently with trying to explain this to a person new to data discipline and found that once again, my answer was much longer and more complicated then I would hope. It seems like such a great concept, but the truth is that everyone has their own way to understand and tackle this problem. With that in mind, I reached out to some of the smartest people I know to see how they tackle the issue. The specific problem I asked about was explaining the difference and contradictory nature of revenue attribution and revenue generation.
Not everyone agrees on the issue or how to express is, and that is why it is so difficult for some, especially those that don’t deal with it on a daily basis. It takes many great voices to find the tools that enable anyone to really correctly tackle large complex issues.
Below are a few of the answers that I was able to gather:
Brent Dykes – Author of Web Analytics Action Hero and general analytics guru -
For example, if my site’s conversion rate is 10%, you’d think that would be great. In the back of your mind, you may remember reading somewhere that the average conversion rate for most sites between 2–3% so 10% sounds fantastic. However, when we start to add context and perform comparisons this number may end up sounding less appealing. What if my site’s conversion rate last year was 15% compared to today’s 10%? What if similar country sites in my organization have 20% conversion rates? What if my closest industry peers recently shared in a media article that they have average conversion rates of 30%? Now the 10% conversion rate doesn’t sound as good.
A rate simply provides us with a number, and what we do with the measure is what adds value. When we analyze what’s happening with the conversion rate, we can determine how to create more value or stop value leakages. Through testing we can confirm what we found in our analysis (correlation vs. causation) before making wholesale changes. It’s important to use the right rates or metrics, but the numbers without any context or comparisons are meaningless. Value only comes from understanding the rates and making changes to improve them over time.
Russell Lewis – Optimization Consultant
You have two QB’s to play. One has a higher completion rate than the other. This rate indicates that he should have a high predicted score when it comes to game time. When you decide to play him, he falls flat on his face. This rate did not give you the value of what his performance is, it just showed what he has done in the past in regards to completed passes to attempted passes. The value of what he actually did is seen when put in comparison to the QB on the bench that had the 10 additional points needed to win the game for the week. Without the comparison to the other QB and the current matchup, we would have no value.
When trying to determine revenue generating sources, I have always relied on a less granular outlook. Rather than saying “this email message generated $X,” stepping back and saying “email campaigns drove $X, while SEO drove $X”. To get much more granular than that and you begin speculating too much about human nature, which is anything but reliable.
To me that is when you get into the psychology of it, and it gets too nit-picky. I think broadly if you are trying to determine whether to put ad dollars in email or SEO it can help…but when you start saying “well, if we put out an email with this call to action, it will generate $X in return” you have a problem.
To me it is a gross misuse of the scientific method…you almost need to look at the control group and see what they are doing before you can determine anything. No one looks at the visitors not associated with a campaign…maybe people on the site just buy stuff on their own.
Jared Lees – Business Consulting Manager
• Revenue allocation – similar to attribution or correlation. Assigning credit to an activity. The amount of credit could depend on the business rules or attribution model you want to do.
• Revenue generation – total revenue acquired from a singular action. There could be other actions that influenced it, but we aren’t counting that here.
Rhett Norton – Senior Retail Consultant & Team Lead
I think the best thing that helps explain these types of situations is explaining causation and correlation.
This made me think of the jobs report on the economy – unemployment rate is 8%, which there is not really anything we can do with that, it is just a number/rate. Lots of people like to look at different sectors and pretend we know what is going on when they can say that growth increased in the technology sector—this is similar to the page participation example above. Again, there isn’t anything you can do with that, it is just a rate. The real question is how do we move the needle, how do we create jobs or what actions make jobs decline.
Derek Tangren – Principal Analytics Consultant -
• Revenue allocation is a method/means to assign success based on certain behavior
• Revenue generation references an action that you are taking in order to invoke a positive change in driving revenue
I would define revenue generation as the action you take and the revenue allocation the means by which you measure the success.
There were many more answers, as you would expect. Some said it didn’t matter because the point is to give executives evidence to continue their agenda, others simplified the situation to simply correlation and causation, and even more didn’t even acknowledge the problem. Most acknowledged that the problem is a major one, but were unable to come up with a simple direct way to convey the message.
Like so much in the online data world, there is no simple answer. Even more, there are as many different agendas and points of views as there are ways to answer the question. Simple answers will always leave you with more questions than answers. How do you deal with this when running your program? Is this the type of battle that you wage, and if not, why? How do you know when you are having the right conversations?