One of the most dif­fi­cult prin­ci­ples to under­stand for many peo­ple in our indus­try is that rate and value is not the same thing. One of the fastest ways for a pro­gram to go astray is to con­fuse one for the other. It is easy for peo­ple to under­stand the need for agree­ing on what you are try­ing to accom­plish, or why they need to have lead­er­ship. It is even easy to talk about the need for effi­ciency and that it is ok to be wrong, but yet even when peo­ple get past that point, they still con­sis­tently miss this crit­i­cal dif­fer­ence. We so des­per­ately want to explain our value to the com­pany, that we con­fuse the value of our actions in an attempt jus­tify our actions. This fun­da­men­tal loss of under­stand leads to a wide range of poor deci­sions and bad under­stand that dra­mat­i­cally lim­its the pos­i­tive impact a test­ing or ana­lyt­ics pro­gram can have.

A rate is sim­ply a ratio or a descrip­tion of actual out­come; it is the same thing as me telling you that I have a $4.23 RPV for a pop­u­la­tion or that I got 5000 con­ver­sions. This is a descrip­tion of past behav­ior, and is sim­ply an out­come, not a descrip­tion of why or how that out­come came to be. Where peo­ple lose focus is that the value, or abil­ity to pos­i­tively or neg­a­tively influ­ence that out­come, is not tied to those gross num­bers. A descrip­tion of rate tells you noth­ing about an indi­vid­ual action, since you are not com­par­ing that out­come, only describ­ing it. Increas­ing your con­ver­sions does not inher­ently cre­ate more rev­enue, nor does the rev­enue by itself reflect pos­i­tive value gen­er­ated by an action. We mea­sure things by say­ing we ran a cam­paign and then we got $3.56, this is not the same as telling you any­thing about the value of that cam­paign. Value would be the dif­fer­ence in run­ning that par­tic­u­lar cam­paign ver­sus not doing any­thing, or run­ning a dif­fer­ent cam­paign. The rate is the end out­come, the value of that action is how much it improved or decreased performance.

Peo­ple are so con­di­tioned to express their con­tri­bu­tion or to explain their value as the out­come of a group. I am respon­si­ble for the prod­uct page, or SEO, or inter­nal cam­paigns so there­fore I must be the sole rea­son for the gen­er­a­tion of that value. Just because your depart­ment or your prod­uct pro­duces 10 mil­lion dol­lars, it does not mean that is rep­re­sen­ta­tive of your value. Value is sim­ply what would have hap­pened if you had done noth­ing, or if you chose a dif­fer­ent route. Value is what we are really talk­ing about in opti­miza­tion, we are dis­cov­er­ing the var­i­ous tools and options that allow us to influ­ence our cur­rent state and improve per­for­mance. We have a way to mea­sure the effi­ciency of dif­fer­ent actions and choose actions based off of a ratio­nal process instead of opin­ion and “expe­ri­ence”. The value of an action is the amount it increases or decreases the bot­tom line, which means your value is the abil­ity to choose the best influ­encers, and avoid the worst ones. Stop defin­ing actions by the rate and instead think in terms of value and you will com­pletely change your view of the world. The ques­tion is never what did it do, but would it have done if I sim­ply stopped to exist, or if we chose any of the other routes avail­able to us. Which one would have gen­er­ated the high­est pos­si­ble outcome?

So where does this cause peo­ple to go astray? The first place is in assum­ing that past behav­ior reflects value, instead of the rate of action. Peo­ple are so used to doing com­pli­cated analy­sis that shows that peo­ple who click on sec­tion Y are worth $3.45. What they are miss­ing is that you are express­ing the rate of rev­enue from peo­ple that click on that sec­tion, not the value of the sec­tion. Using cor­rel­a­tive infor­ma­tion it is impos­si­ble to know what that is really influ­enc­ing, is it pos­i­tive or neg­a­tive? Would you make more or less from hav­ing it? Or what about other alter­na­tives? It is a fun­da­men­tal shift in how you view the world, not focus­ing on what was, but only focus­ing on the influ­ence and cost of changes. Get­ting caught on this def­i­n­i­tion often leads to mis­al­lo­ca­tion of resources and groups hold­ing items sacred that are neg­a­tive to end performance.

This change in view­ing the world also requires also that every per­son accepts that their inputs, skills, and respon­si­bil­i­ties are part of this sys­tem, and that it is rarely going to be a per­fect match between what is best for their group and what is best for the orga­ni­za­tion as a whole. You are not defined by the rate out­put of your respon­si­bil­ity, but what you do with it. What mat­ters is the abil­ity to view every­thing as work­ing together to improve the whole, which neces­si­tates the need to not focus on indi­vid­ual groups, items, or inter­ac­tions. When we are try­ing to gen­er­ate value for the orga­ni­za­tion, and improve our bot­tom line, the least impor­tant item is what does that do to item X, or to sec­tion Y. That infor­ma­tion is rarely mean­ing­ful as it is a sin­gle data point, is always going to cause a cog­ni­tive dis­so­ci­a­tion with what is best for the site and the whole. Get­ting peo­ple to act ratio­nally is not inher­ent to the human con­di­tion, but is vital to get­ting the best results.

The eas­i­est way to prove this sim­ple dis­so­ci­a­tion between rate and value with test­ing is to do an inclusion/exclusion test and sim­ply remove each item one at a time. If you know the rates before hand, or if you believe that an item is worth some value, then it would mean that you would drop that entire value when you remove the item. In real­ity, you will find lit­tle con­nec­tion between that cor­rel­a­tive value and out­comes, and will be shocked by how often you find things that you thought were valu­able, but that turn out to be neg­a­tive to the total page performance.

Test­ing is an amaz­ing tool that allows you the abil­ity to see the value of items. It is not very use­ful for the rate of out­comes, since we are try­ing to com­pare out­comes, but it gives you so much more insight then what you had before. It frees you up to see the world dif­fer­ent and to tackle real prob­lems that you could never tackle before. Under­stand­ing what infor­ma­tion is telling you, what it isn’t, and how best to lever­age dif­fer­ent types of infor­ma­tion together is what changes myth to real­ity in your use of data. In order to start down that path, you must first deeply under­stand the dif­fer­ence between rate and value, and under­stand that your job is not to focus on rate, but instead to dis­cover value.