The human mind is a funny thing. We can be aware of all our own faults, and oth­ers, and yet when it comes to stop­ping our­selves from falling down the many holes that we cre­ate for our­selves, we find it much eas­ier to see the same mis­take by oth­ers then in our­selves. In the next bias I want to tackle is Observer-Expectancy Effect, or “when a researcher expects a given result and there­fore uncon­sciously manip­u­lates an exper­i­ment or mis­in­ter­prets data in order to find it.” Like its sib­ling, con­gru­ence bias, Observer-Expectancy Effect impacts what we test, but it even more fully impacts what con­clu­sions we arrive at. It is about the entire phe­nom­e­non of hear­ing what you want to hear, or using data to only jus­tify what you already wanted to do.

This sin­is­ter worm pops its head up in all types of places, whether it is in exper­i­ment design, using data to jus­tify deci­sions, sales pitches, or even in just our own view of our impact on the world. How much of just reg­u­lar mar­ket­ing is telling you what you want to hear? Yet, we lose focus that we are just as sus­cep­ti­ble to those mes­sages when we are try­ing to prove our­selves right.

What is impor­tant is not as much what the prob­lem is, but how best to “fix” it. How do you insure that you are look­ing at data and get­ting the func­tional result that is best, and not just let­ting your own psy­che lead you down its pre­de­ter­mined path. The trick is to think in terms of the null assump­tion. The chal­lenge is to always assume that you are wrong, and to look at the inverse of all your “Expe­ri­ence”; chal­lenge your­self to think in terms of “what if that wasn’t even there” or “what if we did the exact oppo­site”? Mak­ing sure that you are try­ing to prove the inverse, that you are wrong and you will sud­denly have a much deeper under­stand­ing into the real impact of the out­comes that you are cham­pi­oning. When you try to prove you are right, you will find con­fir­ma­tion, just as when you try to prove you are wrong, you will also come to that con­clu­sion. You have to be will­ing to be “wrong” in order to get a bet­ter out­come. Remem­ber that when you are wrong, you get the ben­e­fit of the increased results, and you have learned something.

So what does this look like in the real world? Every time you decide that you are going to go down a path, you will intrin­si­cally want to prove to your­self and oth­ers that what you are doing is valu­able. The most com­mon exam­ple of this is in the quest for per­son­al­iza­tion, where we get so caught up in prov­ing we can tar­get to groups that we for­get to mea­sure the real impact of this deci­sion. We for­get that the same per­son can be looked at a thou­sand dif­fer­ent ways, so when we choose to pick one, that we fail to mea­sure it against the other alter­na­tives. The num­ber of groups that have cham­pi­oned tar­get­ing to some minute seg­ment, who when you look deeper into the num­bers and find that tar­get­ing to browser or time of day would have mag­ni­tudes of greater impact, is legion.

The sim­plest way to test this is to make sure that all of your eval­u­a­tions, cor­rel­a­tive, causal, or qual­i­ta­tive, include the null assump­tion. What hap­pens if I serve the same changed con­tent to every­one? Or what hap­pens if serve tar­geted con­tent to fire­fox users instead? Despite the con­stant ban­ter and my belief that a per­son­al­ized expe­ri­ence is a good thing, what do I really see from my exe­cu­tion? What about if we tar­get to the groups that don’t show dif­fer­ent behav­ior in our analy­sis? Keep decon­struct­ing ideas and keep try­ing to find ways to break all the rules, and you will find them. Even bet­ter, those are the moments where you truly learn and where you truly get value that you would not have got­ten from just tak­ing straight to the action.

This is not just a prob­lem with ana­lyt­ics; it plays out with any sort of analy­sis, espe­cially A/B test­ing. So many groups make the mis­take of just test­ing their hypoth­e­sis against another, which they fail to see the big­ger pic­ture. Hypoth­e­sis test­ing is designed to be absolutely sure of the valid­ity of a sin­gle idea, not to com­pare other ideas or to reach any con­clu­sion at a mean­ing­ful speed. It is the end point of a long dis­ci­plined process, not the start­ing point where so many want to lever­age it.

The final com­mon way this plays out is when we mis­take a rate of an action with the value of the action. We get so caught up in want­ing to believe some lin­ear rela­tion between items, that hav­ing a great pro­mo­tion and get­ting more peo­ple to click on it equals more value, that we fail to mea­sure the end goal. We mis­take the con­cept we are try­ing to prop­a­gate with the end goal, assum­ing that if we are suc­cess­ful in push­ing towards a desired action, that we have accom­plished our end goal. Hav­ing run on aver­age 30 tests a week with dif­fer­ent groups over the last 7 years, I can tell you that from my own expe­ri­ence, the times when this plays out in the real world I can count on 1 hand.

So much analy­sis loses all value because we are pre-wired to just accept the first thing we find, or to find data to con­firm what we want to believe, or that we then send out that data to oth­ers to prove our point and ignore the larger world. We are so wired to want to think we are mak­ing a dif­fer­ence that we con­stantly fail to dis­cover if this is true. Be bet­ter then what you are wired to believe and force your­self to think in terms of the null assump­tion. Think in terms of pur­posely look­ing at the oppo­site of what you are try­ing to prove or what you believe. The worst case is that you have spent a few more moments and con­firmed, truly, what you believe. The best case sce­nario is that you have now changed your world view and got­ten a bet­ter result, one that is not arrived at sim­ply because you expected to arrive at that point.

0 comments