There are many chal­lenges for any­one enter­ing a new field of study or a new dis­ci­pline. We are all com­ing into any new con­cept with all of our pre­vi­ous held knowl­edge and pre­vi­ous held beliefs fil­ter­ing and chang­ing how we view the new thing before us. Some choose to make it fit their world view, oth­ers dis­miss it from fear, and oth­ers look for how it can change their cur­rent world view. Usu­ally in these sit­u­a­tions I quote Sher­lock Holmes, “It is a cap­i­tal mis­take to the­o­rize before one has data. Insen­si­bly one begins to twist facts to suit the­o­ries, instead of the­o­ries to suit facts.” Noth­ing rep­re­sents this chal­lenge more in online mar­ket­ing then the dif­fer­ences between ana­lyt­ics and opti­miza­tion, and noth­ing rep­re­sents that strug­gle more than the debate about visit based mea­sure­ment ver­sus vis­i­tor based measurement.

The debate about should some­one use vis­its, impres­sions, or vis­i­tor basis for analy­sis is a per­fect exam­ple of this prob­lem, as it is not as sim­ple as always use one or the other. When you are doing ana­lyt­ics, usu­ally vis­its are the best way to look at data. When you are doing opti­miza­tion, there is never a time where vis­its would present to you more rel­e­vant infor­ma­tion then using a vis­i­tor based view of the data.

Ana­lyt­ics = Visit
Opti­miza­tion = Visitor

The only pos­si­ble excep­tions are when you are using adap­tive learn­ing tools. While the rules can be sim­ple, a deep under­stand­ing of the way presents many other oppor­tu­ni­ties to improve your over­all data usage and value derived from every action.

Since most peo­ple read­ing this start in an ana­lyt­ics back­ground, let’s look at what works best in that envi­ron­ment. Ana­lyt­ics is a sin­gle data set cor­rel­a­tive data met­ric sys­tem, which is a long way of say, it counts things on a con­sis­tent basis and only one set of data, even if that data has many dif­fer­ent dimen­sions. You are only record­ing what was, not what could or should be. In that envi­ron­ment, you have to look at data in some very par­tic­u­lar ways. The first amongst those is a very tight con­trol on accu­racy, since in many cases the use of that data is to rep­re­sent what the busi­ness did, and to hope­fully make pre­dic­tions about the future.

It is also impor­tant that you are con­sis­tent with how you mea­sure and that you look at things in a com­mon basis. Because most peo­ple are com­fort­able look­ing at a day or shorter term basis, this means the eas­i­est method is going to be a visit. It is works great because you are try­ing to look at inter­ac­tions and to mea­sure in a raw count of things that did hap­pen, e.g. how many con­ver­sions, or how many peo­ple came from SEO. In those cases, a raw count in a cor­rel­a­tive area is going to be best rep­re­sented using a visit basis, since it mit­i­gates lost data (though it is not a mas­sive amount) and it best reflects the com­mon basis that peo­ple look at data.

In the world of opti­miza­tion how­ever, you have a com­pletely dif­fer­ent usage and type of data. In opti­miza­tion we are look­ing at a sin­gle com­par­a­tive data point, and try­ing to rep­re­sent an entire dif­fer­ent mea­sure, which is influ­ence on behav­ior over time. It doesn’t mat­ter if your site changes once a year or once an hour, or if your buy­ing cycle is 1 visit or 180 days, all of those things are irrel­e­vant to the fact that you are influ­enc­ing a pop­u­la­tion over time. Because behav­ior is defined as influ­ence on a pop­u­la­tion, and because we are look­ing com­par­a­tively over time, the mea­sure­ment tech­niques used in ana­lyt­ics need to be rethought. Any con­cern about accu­racy, past a sim­ple point, become far less impor­tant than a mea­sure of pre­ci­sion (con­sis­tency of data col­lec­tion) since all error derived is going to be equally dis­trib­uted. It doesn’t mat­ter if the com­mon basis is $4.50 or $487.62, what mat­ters is the rel­a­tive change based on the con­trolled fac­tor. It is also impor­tant that we are focus­ing far more on the influ­ence then the raw count, which means we are really talk­ing about the behav­ior of the population.

In ana­lyt­ics you are think­ing in terms of, what was the count of the out­come (rate) as opposed to in opti­miza­tion the focus is on what was the influ­ence (value). To really under­stand opti­miza­tion, you have to under­stand that all groups start with a stan­dard propen­sity of action which is rep­re­sented by your con­trol group. If you do noth­ing the peo­ple com­ing to your site, peo­ple in all stages and all types of inter­ac­tion, mea­sure up to one stan­dard mea­sure across your site (though all mea­sure­ment sys­tems do have inter­nal vari­ance in a small degree). Since we are mea­sur­ing not what the propen­sity of action is but what are abil­ity to pos­i­tively or neg­a­tively influ­ence it is, we need to think in terms of report­ing based on vis­i­tors and based on the change (lift) and not the raw count.

You also have the case of time, where we need to mea­sure total impact over time. While it is cor­rect that every time a vis­i­tor hits your site you have a chance to influ­ence them, it is impor­tant to remem­ber that the exist­ing propen­sity of action mea­sure­ment already accounts for this. What we are look­ing for is a sim­ple mea­sure of what did we accom­plish by in terms of get­ting them to spend more. This means that we have to think in terms of both long and short term behav­ior. Some peo­ple will pur­chase today, some 3 vis­its later, but all of that is part of stan­dard busi­ness as usual. It is incred­i­bly easy to have sce­nar­ios where you get more imme­di­ate actions but less long term actions. This means that on a daily basis you might see a short term spike, but for the busi­ness over­all you are going to be mak­ing actu­ally less rev­enue. This pos­si­bil­ity cre­ates two pos­si­ble mea­sure­ment scenarios:

1) There is no dif­fer­ence between short term and long term behav­ior, mean­ing the short term spike con­tin­ues through and is pos­i­tive also in the long term. In this sce­nario the only way to know that is to look at the long term.

2) There is a dif­fer­ence and short and long term behav­iors dif­fer and we are get­ting a dif­fer­ent out­come by look­ing at the vis­i­tor met­ric over time. In this sce­nario the only pos­i­tive out­come for the busi­ness is the vis­i­tor based met­ric view.

In both cases the vis­i­tor based met­ric view gives us the full pic­ture of what is good for the busi­ness, while the visit based met­ric sys­tem either has no addi­tional value or a neg­a­tive value by reach­ing a false con­clu­sion. In either case the only mea­sure that adds value and gives us a full pic­ture is the vis­i­tor based view of the world. We have a case where vis­i­tor is both the most com­plete view, no mat­ter the sit­u­a­tion, but the only one that can give you a ratio­nal view of the impact of a change. To top it off, the choice to only look at the shorter win­dow cre­ates a dis­tri­b­u­tion bias, by valu­ing short term behav­ior over long term behav­ior, which may cre­ate ques­tions into the rel­e­vance of the data used to make any conclusion.

The vis­i­tor vs. visit based view of the world is just one of many mas­sive dif­fer­ences that reduce the value derived from opti­miza­tion if not under­stood or not eval­u­ated as a sep­a­rate dis­ci­pline. Because it is so easy to ratio­nal­ize stick­ing with what is com­fort­able, it is com­mon to find this mas­sive weak­ness being prop­a­gated through­out orga­ni­za­tions with no mea­sure of what the cost really is. While not as dam­ag­ing as oth­ers, like not hav­ing a sin­gle suc­cess met­ric or not under­stand­ing vari­ance, it is vital that you are think­ing about visit and vis­i­tor based data as attached the end goal and not as a sin­gle answer to everything.

In the end, the debate about which ver­sion to use is not really one about vis­its or vis­i­tors, there are clear rea­sons to choose vis­its for ana­lyt­ics and vis­i­tor for opti­miza­tion. The real chal­lenge is if you and your orga­ni­za­tion under­stand the dif­fer­ent data dis­ci­plines that are being lever­aged. If you con­stantly look for dif­fer­ent ways to think about each action you will find new and bet­ter ways to improve value, if you fail to do so you will cause dam­age through­out your orga­ni­za­tion and will not even know you are doing it.

0 comments