One of my favorite online blogs is You are Not So Smart. I find it fas­ci­nat­ing the fal­lac­ies that make up our day to day lives, and find that the num­ber one dri­ver of chang­ing client cul­tures is to tackle and teach lessons that help them chal­lenge or break some of these fallacies.

The first of these biases is CONGRUENCE BIAS, or “the ten­dency to test hypothe­ses exclu­sively through direct test­ing, in con­trast to tests of pos­si­ble alter­na­tive hypothe­ses”, or oth­er­wise known as, try­ing to prove myself right. You see this all the time in test­ing groups, as they test only 1 or 2 things that are very sim­i­lar, then get a small win and claim it is the great­est thing since sliced bread. You also see this when they know they are going to test tar­get­ing to return users, or chang­ing the prod­uct image on their land­ing pages, or any pre­de­ter­mined pri­or­ity of ele­ments or changes. We are all guilty of this, and it takes dis­ci­pline and ded­i­ca­tion to not let it run your pro­gram or your life.

There are entire sub indus­tries who feed and live off of this fal­lacy, like sites that show win­ning tests, many “top” blogs, or peo­ple or agen­cies who sell “we have all the answers”. Even inter­nally, how often have you sat around hav­ing “I feel/I believe” con­ver­sa­tions to deter­mine what the best course of action is (and as a side, why do peo­ple think those two state­ments mean some­thing dif­fer­ent in that con­text?). You also see the same thing when you have groups who all they want to do is talk about tests, not about their pro­gram. It means that they are so caught up on try­ing to prove them­selves right that they are miss­ing the for­est for the leaves.

It is fas­ci­nat­ing how often the groups that get some of the worst ROI for their test­ing are the ones who cham­pion and talk about it the most, and the rea­son for it is this bias. The peo­ple in charge are try­ing to make them­selves look good (I had the right answer) instead of find­ing the best answer. The answer is sim­ple… know you are wrong… even if you have the per­fect “best” answer today, you are going to be wrong in a num­ber of months.  If you don’t have the con­text for the result you have, or if you don’t hear the con­text for the result that some­one is push­ing on you, then you do not know any­thing about the value of the outcome.

If you want to make sure that you are both get­ting the most value, and that you are let­ting a pro­gram learn and grow to the best con­clu­sion, the choice is sim­ple, chal­lenge this bias. Include the null assump­tion, include every fea­si­ble alter­na­tive, even if you think it won’t win (there is an almost inverse cor­re­la­tion between what peo­ple think will win and what will win). Chal­lenge that just because one per­son out there got a 10% lift doing some­thing, that it is the best thing for you? How do you know how many other peo­ple failed mis­er­ably doing it? How do you know that even if they got the 10% that they couldn’t have got­ten 15 or 20% by doing some­thing sim­pler and with less effort?

Chal­lenge your­self… Chal­lenge how you think and chal­lenge how oth­ers think and you will find the next “truth”. Let the real­ity of the data (the CASUAL data) tell you where to go and what to do, don’t let hubris or pop­u­lar opin­ion tell you the value of something.

The sys­tem is what gets you val­ues any­ways, not any indi­vid­ual an action.

Dave Lloyd
Dave Lloyd

Great reminder Andrew. I'll be sharing this at my next team meeting.