I was hav­ing a con­ver­sa­tion with a col­league about the best way to assist a cus­tomer with their 2012 plan­ning, and it brought up the third bias I deal with on a com­mon basis, Hyper­bolic Dis­count­ing or “the ten­dency for peo­ple to have a stronger pref­er­ence for more imme­di­ate pay­offs rel­a­tive to later pay­offs, where the ten­dency increases the closer to the present both pay­offs are.” Besides being the basis for the entire credit card indus­try, in opti­miza­tion this mostly comes out in terms of bet­ter ver­sus best testing.

When groups decide on a roadmap, or the list of tests that they mis­tak­enly refer to as a roadmap, they will often pri­or­i­tize them by what is most on top of people’s minds, or what their boss wants. You have all been sit­ting around and think that chang­ing the but­ton to Buy instead of Buy Now, or you really want to change the back­ground on your pro­mo­tional images on your front door, or the copy on your land­ing page, you think it needs to say X… All of this leads to the want to test your the­ory and prove your­self right. You shut down, ignore dis­ci­pline, and try to see if that idea is bet­ter than the other. The entire con­cept of hypoth­e­sis test­ing leads to this fail­ure of human cog­ni­tion. It works if the only goal is to prove a sin­gle point, but it is really inef­fi­cient if the goal is mea­sure the rel­a­tive value of an action. We for­get that we are deal­ing with more then the day to day issues in front of us, and we try to solve a prob­lem today instead of push­ing to get the most from our entire pro­gram. Even worse, it leads to the pri­or­i­ti­za­tion of these tests over more effi­cient and dis­ci­pline based tests because of the imme­di­ate pay­off in the “I am right” reward they offer.

The rea­son that this fails is because the focus is on that short term gain, we often think it will take too much effort, both resource and espe­cially polit­i­cally to try to learn in our efforts. Instead, you end up down the path of sim­ply try­ing to fig­ure out if ver­sion B is bet­ter than ver­sion A. It isn’t about fig­ur­ing out what you should be doing, it is about try­ing to prove your idea bet­ter then the other per­sons idea. In some cases, groups will add a few smaller tweaked ver­sions of A or B (which is still bet­ter), but end of the day, your test ends up answer­ing, “Which is bet­ter, ver­sion A or ver­sion B”. If the con­cept is bad, it is an inef­fi­cient test. If the sec­tion you are opti­miza­tion is not as impor­tant, ver­sion B can be mas­sively bet­ter then ver­sion A, but still only as valu­able as the least “bet­ter” ver­sion of a dif­fer­ent sec­tion of the same page (or a dif­fer­ent type of change to the same item), it is still an inef­fi­cient test. It is far more valu­able to know that if I spend $5, I can get $15, $7, $20, or $30 then to just know I got $7 and be happy with it. True opti­miza­tion is to fig­ure out the best path, not to just mea­sure the one that you are already on. Yes, you get a result, but it is a really inef­fi­cient result, and often leads to fur­ther inef­fi­cient results as you con­tinue down that path. Even worse is when you try to force a MVT to just throw mul­ti­ple bet­ter test together and sim­ply increase the speed of your sub opti­mal out­come. There is no way to make this type of test­ing effi­cient, unless you get ahead of the prob­lem and can decon­struct the ques­tion of the test.

Best test­ing takes a dif­fer­ent approach, it asks much more fun­da­men­tal ques­tions, such as “Does my copy mat­ter the most on the page?”, or “which fac­tor of the but­ton is most influ­en­tial?”, “what are the fea­si­ble options”, or even “does chang­ing the but­ton, is that the best place to put my resources?”. Best test­ing is about fig­ur­ing out where the best places are to focus your energy and the best fea­si­ble alter­na­tive; democ­ra­tiz­ing the entire process so that your out­comes are less biased by any indi­vid­ual or idea. Best test­ing becomes about the sys­tem you have in place to make deci­sions, and not about any indi­vid­ual idea that goes into it. Any sys­tem is only as good as the qual­ity that goes into it, but it designed to max­i­mize your returns that come out of that sys­tem. It also makes you decon­struct your ques­tions, one of the sin­gle most impor­tant skills for a suc­cess­ful pro­gram, and chal­lenge core assump­tions and biases that you have. The key is to fig­ure out what places and what types of changes are BEST rel­a­tive to each other, so that you can align resources and inter­nal thought towards those points, while elim­i­nat­ing waste on things that do not mat­ter. The act of valu­ing pos­si­ble actions against each other with causal infor­ma­tion is the sin­gle great­est way to max­i­mize resources, not use them unnec­es­sar­ily. It chal­lenges you to look at the page and user expe­ri­ence as a holis­tic item, and that each com­po­nent is part of that process, and then to weigh the value of each one AGAINST each other. It isn’t about your idea, or even any indi­vid­ual idea any more, it is about cre­at­ing a sys­tem by which you can learn and fig­ure out where to focus your energy to get even bet­ter con­cepts and even bet­ter out­comes. Who cares if the copy improves the page if it only does it at 1/10th the scale of chang­ing the back­ground, or the main image, or that small sec­tion on the page that you never expected?

What is amaz­ing is that you are left with two options from the BEST testing:

1) You were cor­rect — In which case you have con­fir­ma­tion and you still get the same lift.

2) You are wrong — You learn valu­able infor­ma­tion, and you get MORE lift then what you had before. Even bet­ter, it might send you down a path that you weren’t even con­sid­er­ing before.

What hap­pens when you are stuck doing bet­ter test­ing, is that you get the short term return, both on your idea, but also polit­i­cally by show­ing that you were cor­rect. You are ignor­ing the long term oppor­tu­nity cost of fig­ur­ing out what mat­ters most and build­ing off of some­thing. It gives you a shiny object that you get to show off to any­one that will lis­ten, but the real ques­tion is what are you giv­ing up going down that path?

The irony of this is that every­one always thinks that that will take too much time or resources, when for a large major­ity of sites out there, it is almost always the exact oppo­site. Being effi­cient means that you are test­ing based on your cur­rent resources and the way to max­i­mize speed, where bet­ter test­ing often leads to forc­ing an overly tech­ni­cal solu­tion in order to exe­cute the one idea that you are will­ing to con­sider. What is true how­ever is that you have to shift how peo­ple think, and chal­lenge them to under­stand some new dis­ci­plines, in order for them to accept and exe­cute in this way. It takes a cul­ture and a per­son will­ing to be “wrong” and one who is will­ing to let go of their ego for the sake of the site. To para­phrase Kathryn Schulz, “you are already wrong, the dif­fer­ence is sim­ply that you will know that you are wrong”.

What you get out of your opti­miza­tion pro­gram is all about the sys­tem you put in place and how much you are will­ing to chal­lenge your­self and oth­ers to find the best thing you can be doing. It is easy to want to take the imme­di­ate pay­out of being right or prov­ing a HiPPO right, but at the end of the day, you will always be bet­ter off if you focus on the dis­ci­plines that make you suc­cess­ful. The best thing that test­ing can do is chal­lenge or elim­i­nate human biases from the plate. It can cre­ate an equal play­ing field so that you can cor­rectly know the causal value of an item, and to be able to mea­sure the EFFICIENCY of improv­ing it. It allows all ideas to be mea­sured against them­selves, and more impor­tantly, it stops groups from spend­ing hours talk­ing about ways to improve things that don’t mat­ter, while teach­ing what does mat­ter and how best to impact it. It doesn’t elim­i­nate the value of good input; it just fil­ters it and refines it so that it is spent on the cor­rect things, not just what per­son A thinks is the most impor­tant thing.