One of my great strug­gles in the entire data world is to get peo­ple to under­stand the dif­fer­ence in rate and value. It seems like this prob­lem has a thou­sand dif­fer­ent faces, yet it can be extremely dif­fi­cult to find the right way to cor­rect the mis­con­cep­tions of any par­tic­u­lar case. Peo­ple are con­stantly try­ing to abuse data to show that they pro­vided value or that some­thing is directly tied to an out­come, despite the fact that the data itself can not in any way tell you this fact.

I was faced recently with try­ing to explain this to a per­son new to data dis­ci­pline and found that once again, my answer was much longer and more com­pli­cated then I would hope. It seems like such a great con­cept, but the truth is that every­one has their own way to under­stand and tackle this prob­lem. With that in mind, I reached out to some of the smartest peo­ple I know to see how they tackle the issue. The spe­cific prob­lem I asked about was explain­ing the dif­fer­ence and con­tra­dic­tory nature of rev­enue attri­bu­tion and rev­enue generation.

Not every­one agrees on the issue or how to express is, and that is why it is so dif­fi­cult for some, espe­cially those that don’t deal with it on a daily basis. It takes many great voices to find the tools that enable any­one to really cor­rectly tackle large com­plex issues.

Below are a few of the answers that I was able to gather:

Brent Dykes – Author of Web Ana­lyt­ics Action Hero and gen­eral ana­lyt­ics guru -

A rate is sim­ply a cal­cu­lated met­ric. We use rates to mea­sure all kinds of things such as the bounce rate of a land­ing page or con­ver­sion rate of a check­out process. In order to get value from rates we need both con­text and com­par­isons. On its own, a rate doesn’t tell us any­thing useful.

For exam­ple, if my site’s con­ver­sion rate is 10%, you’d think that would be great. In the back of your mind, you may remem­ber read­ing some­where that the aver­age con­ver­sion rate for most sites between 2–3% so 10% sounds fan­tas­tic. How­ever, when we start to add con­text and per­form com­par­isons this num­ber may end up sound­ing less appeal­ing. What if my site’s con­ver­sion rate last year was 15% com­pared to today’s 10%? What if sim­i­lar coun­try sites in my orga­ni­za­tion have 20% con­ver­sion rates? What if my clos­est indus­try peers recently shared in a media arti­cle that they have aver­age con­ver­sion rates of 30%? Now the 10% con­ver­sion rate doesn’t sound as good.

A rate sim­ply pro­vides us with a num­ber, and what we do with the mea­sure is what adds value. When we ana­lyze what’s hap­pen­ing with the con­ver­sion rate, we can deter­mine how to cre­ate more value or stop value leak­ages. Through test­ing we can con­firm what we found in our analy­sis (cor­re­la­tion vs. cau­sa­tion) before mak­ing whole­sale changes. It’s impor­tant to use the right rates or met­rics, but the num­bers with­out any con­text or com­par­isons are mean­ing­less. Value only comes from under­stand­ing the rates and mak­ing changes to improve them over time.

Rus­sell Lewis – Opti­miza­tion Consultant

Here is one that spawns from my lat­est fan­tasy foot­ball win.

You have two QB’s to play. One has a higher com­ple­tion rate than the other. This rate indi­cates that he should have a high pre­dicted score when it comes to game time. When you decide to play him, he falls flat on his face. This rate did not give you the value of what his per­for­mance is, it just showed what he has done in the past in regards to com­pleted passes to attempted passes. The value of what he actu­ally did is seen when put in com­par­i­son to the QB on the bench that had the 10 addi­tional points needed to win the game for the week. With­out the com­par­i­son to the other QB and the cur­rent matchup, we would have no value.

Anony­mous –

To me, rev­enue allo­ca­tion has always been a method for rank­ing per­for­mance in much the same way page views or vis­its are. It gives you some­thing to sort by, and that’s about it. Not to men­tion that depend­ing on the type of allo­ca­tion you are using you may be inflat­ing your total rev­enue any­way, so it inher­ently is not a reli­able method of deter­min­ing rev­enue gen­er­at­ing sources.

When try­ing to deter­mine rev­enue gen­er­at­ing sources, I have always relied on a less gran­u­lar out­look. Rather than say­ing “this email mes­sage gen­er­ated $X,” step­ping back and say­ing “email cam­paigns drove $X, while SEO drove $X”. To get much more gran­u­lar than that and you begin spec­u­lat­ing too much about human nature, which is any­thing but reliable.

To me that is when you get into the psy­chol­ogy of it, and it gets too nit-picky. I think broadly if you are try­ing to deter­mine whether to put ad dol­lars in email or SEO it can help…but when you start say­ing “well, if we put out an email with this call to action, it will gen­er­ate $X in return” you have a problem.

To me it is a gross mis­use of the sci­en­tific method…you almost need to look at the con­trol group and see what they are doing before you can deter­mine any­thing. No one looks at the vis­i­tors not asso­ci­ated with a campaign…maybe peo­ple on the site just buy stuff on their own.

Jared Lees – Busi­ness Con­sult­ing Manager

Here is my short answer:

• Rev­enue allo­ca­tion – sim­i­lar to attri­bu­tion or cor­re­la­tion. Assign­ing credit to an activ­ity. The amount of credit could depend on the busi­ness rules or attri­bu­tion model you want to do.

• Rev­enue gen­er­a­tion – total rev­enue acquired from a sin­gu­lar action. There could be other actions that influ­enced it, but we aren’t count­ing that here.

Rhett Nor­ton – Senior Retail Con­sul­tant & Team Lead

I think what the per­son was say­ing some­thing like this, “I looked at a spe­cific chan­nel and it said it gen­er­ated $4.50 worth of rev­enue.” And then you would say “Who cares what the rate is, we need to find what impacts true value and actu­ally changes rev­enue since that num­ber is just a rate.”

I think the best thing that helps explain these types of sit­u­a­tions is explain­ing cau­sa­tion and correlation.

This made me think of the jobs report on the econ­omy – unem­ploy­ment rate is 8%, which there is not really any­thing we can do with that, it is just a number/rate. Lots of peo­ple like to look at dif­fer­ent sec­tors and pre­tend we know what is going on when they can say that growth increased in the tech­nol­ogy sector—this is sim­i­lar to the page par­tic­i­pa­tion exam­ple above. Again, there isn’t any­thing you can do with that, it is just a rate. The real ques­tion is how do we move the nee­dle, how do we cre­ate jobs or what actions make jobs decline.

Derek Tan­gren – Prin­ci­pal Ana­lyt­ics Consultant -

I’d describe the two as follows:

• Rev­enue allo­ca­tion is a method/means to assign suc­cess based on cer­tain behav­ior
• Rev­enue gen­er­a­tion ref­er­ences an action that you are tak­ing in order to invoke a pos­i­tive change in dri­ving revenue

I would define rev­enue gen­er­a­tion as the action you take and the rev­enue allo­ca­tion the means by which you mea­sure the success.

There were many more answers, as you would expect. Some said it didn’t mat­ter because the point is to give exec­u­tives evi­dence to con­tinue their agenda, oth­ers sim­pli­fied the sit­u­a­tion to sim­ply cor­re­la­tion and cau­sa­tion, and even more didn’t even acknowl­edge the prob­lem. Most acknowl­edged that the prob­lem is a major one, but were unable to come up with a sim­ple direct way to con­vey the message.

Like so much in the online data world, there is no sim­ple answer. Even more, there are as many dif­fer­ent agen­das and points of views as there are ways to answer the ques­tion. Sim­ple answers will always leave you with more ques­tions than answers. How do you deal with this when run­ning your pro­gram? Is this the type of bat­tle that you wage, and if not, why? How do you know when you are hav­ing the right conversations?

0 comments