In the first part of our look at advanced par­a­digms, I focused on the com­plex inter­play of test­ing and other parts of your orga­ni­za­tion. As test­ing grows, it starts to inter­act on a nearly daily basis with every part of your orga­ni­za­tion. If you look at the evo­lu­tion that we have taken, going from the very fun­da­men­tal build­ing blocks of a test­ing pro­gram, to the ways we look at tests and test­ing, and finally to the com­plex inter­ac­tions of test­ing into every­thing, we have shifted the impor­tance and the value that test­ing brings. The final stage of evo­lu­tion is to start eval­u­at­ing your own core beliefs of even what is a test­ing pro­gram, data, and even how we view the world. It is easy to chal­lenge oth­ers to grow, but the most dif­fi­cult and most reward­ing changes always start from within. If the evo­lu­tion starts with get­ting peo­ple to align, it ends with chang­ing our fun­da­men­tal beliefs about data. We have to ask extremely dif­fi­cult ques­tions and chal­lenge our own inter­ac­tions, break­ing down our beliefs and rebuild­ing them to strengthen and evolve.

To that end, here is the final look at advanced opti­miza­tion paradigms:

No More Focus­ing on Test Ideas –

If we view opti­miza­tion as a dis­ci­pline, one that never starts and never ends, one that is about the con­stant chang­ing and learn­ing of a user expe­ri­ence, then there is no longer any need for indi­vid­ual test ideas. An idea nat­u­rally has a start and an end, as any hypoth­e­sis comes from a belief in a spe­cific solu­tion to an exist­ing prob­lem. Peo­ple get so caught on their idea, be it from their own expe­ri­ence, some piece of data that they just know means they have the solu­tion to all your prob­lems, or just “best prac­tices” that their brains shut down and they stop try­ing to find the best answer. The prob­lem here is the entire process leads to a myopia that gives us the right to stop, the abil­ity to prove our­selves “right” and the nat­ural affin­ity towards a set path.

If the focus is no longer on test ideas, how­ever, then the sys­tem is what you focus on. If we are instead treat­ing things as a cycle: explore, learn, ideate on all fea­si­ble alter­na­tives, exe­cute, learn, repeat, then there really is no indi­vid­ual cam­paign or test. The path is never about what you think will win, or what you want to tackle, but instead only on where the casual data leads you and the eval­u­a­tion of all fea­si­ble alter­na­tives. Test ideas become the least impor­tant thing you can dis­cuss, and should be viewed with high lev­els of skep­ti­cism. There is no such thing as a good test idea, only a con­cept that can be bro­ken apart, chal­lenged, and improved. Fear any expert try­ing to tell you a sin­gle great test idea, or any guar­an­teed set of steps to improve your site, as they are only play­ing to your own inse­cu­ri­ties; the real­ity is any sin­gle idea can not hold up to scrutiny. This impacts your own beliefs as much as any other, you have to hold your­self to the same level of scrutiny, not allow­ing what you want to hap­pen to be the path that you go down or the answers you seek. You have to be will­ing to step out­side your own opin­ion and be able to focus on all fea­si­ble alter­na­tives and only what the effi­ciency of changes are; any bias that you allow to limit what you test, be it because of expe­ri­ence or pop­u­lar opin­ion, deval­ues the out­come that you can generate.

The biggest chal­lenge most peo­ple have is the feel­ing of a los­ing of con­trol. We often like to blame other groups for this behav­ior, but by far the most guilty group are ana­lysts who are so busy try­ing to prove a point with their data that they fail to see the larger pic­ture. They so want to prove a path using their ana­lyt­ics that they fail to fac­tor in the need to change to an active form of data acqui­si­tion in order to move for­ward. You have to worry about your own biases before you can stop oth­ers. It is easy for all groups to get focused on what their expe­ri­ence or gut tells them is right, often to poor and inef­fi­cient out­comes. Make it clear that no idea stands alone. Put in place mea­sures to insure that you are not lim­ited to pop­u­lar opin­ion or only what you think or want to win. This often means that you have to pri­or­i­tize resources in ways that you are not doing today, but ulti­mately this is the only way to insure you are get­ting the great­est value and insur­ing your own con­tin­ued edu­ca­tion about what the value of actions are.

Free your­self from the cycle of defend­ing and push­ing every idea, instead cre­at­ing momen­tum and a con­sis­tent pat­tern of action. Every­one is afraid of mov­ing towards the infa­mous 48 shades of blue extent of this path, but the real­ity is it frees you. You no longer need con­sen­sus and you can push the bound­aries of what you try. Once you have got­ten to blue as the most impor­tant ele­ment, you may, depend­ing on your feel­ing for the N-Armed ban­dit prob­lem, want to test out to 48 dif­fer­ent vari­a­tions, but that is not an affront to you. Peo­ple built the sys­tem, fed the sys­tem, and con­trol the sys­tem. Once you get to the point that you know what you need to know, why not let the sys­tem pro­vide the answer for you? The sys­tem is only as valu­able as the peo­ple who feed it, yet we fear the sys­tem and we fear becom­ing lost to the system.

Mov­ing down this path, of avoid­ing indi­vid­ual ideas or about try­ing to find the per­fect solu­tion allows you to re-imagine and recre­ate who and what you are on the fly, with­out mas­sive redesign efforts. It allows you to avoid hold­ing any­thing sacred on the site or about ever wor­ry­ing about the entire con­cept of “right”. The user expe­ri­ence becomes a fluid thing, where the true value of data, your cre­ativ­ity, and the abil­ity to move past your own biases, deter­mines the mag­ni­tudes of growth that you will expe­ri­ence. The true value of the indi­vid­ual is in how well they feed the sys­tem. The sys­tem is only as valu­able as what goes into it and by democ­ra­ti­zat­ing all ideas; by forc­ing the con­ver­sa­tion away from ideas and towards fea­si­ble alter­na­tives, you are giv­ing more value to the cre­ative free­dom of the mem­bers of your organization.

Opti­miza­tion pow­ered Analytics –

Let me pose a the­o­rem to you: Ana­lyt­ics, by itself, is com­pletely worthless.

Let me chal­lenge you by look­ing at the entire cur­rent prac­tice of ana­lyt­ics as noth­ing more than hubris. That the cur­rent use of ana­lyt­ics, espe­cially by those that per­pet­u­ate to be experts, is noth­ing but a newer accepted jus­ti­fi­ca­tion for what you were already going to do or where already think­ing. Every new mis­un­der­stand­ing of mon­ey­ball, or of advanced sta­tis­ti­cal mod­els, is a sales pitch designed to make you feel like you are mak­ing a much larger impact then you really are. This is not to say that ana­lyt­ics can not be pow­er­ful, only that the way that data is abused by the prac­ti­tion­ers of the indus­try to prop­a­gate myths and bad prac­tices is worse then worth­less, it’s inefficient.

Peo­ple have got­ten so lost in their abil­ity to col­lect infor­ma­tion, the speed we can get feed­back, and the need to jus­tify their exis­tence that they never take a moment to ques­tion what can you really get from pure ana­lyt­ics. Num­bers have become the new shield by which we per­suade oth­ers of our “great­ness”, not to actu­ally pro­vide value, but instead using data to tell sto­ries dri­ven by ego and a want to be the one mak­ing the deci­sion. We so want to tar­get a group that we find one that stands out, or we so want to show our value that we tell some­one they are doing some­thing wrong, only to replace their “bad” deci­sion with an equally biased one, tak­ing credit for any result that comes from this use resources. In the rare best case sce­nario with ana­lyt­ics, you are left with prob­a­bil­i­ties and no clear direc­tion, in the worst and most com­mon cases, we are left with biased “insights” pow­ered by every­thing but data.

In real­ity, we are no longer trapped by this use of data, like so many other indus­tries before, because of our abil­ity to inter­act directly and in an effi­cient and speedy man­ner. The data loses all value when we force a path on it or we for­get what it can and is really telling us. Because we are a new field, mostly manned by peo­ple with­out real prac­ti­cal data dis­ci­pline, we allow our own lack of under­stand­ing of the nature of data to allow our own biases to paint a pic­ture that does not exist. There are hun­dreds of agen­cies, groups, and peo­ple who claim to have the newest way to repeat the same types of “analy­sis” with­out any newer insight into the value of that data. There are always new ways to cor­rupt sta­tis­tics or dif­fer­ent analy­sis tech­niques that are used to push an agenda, not to actu­ally pro­vide real value. We are so busy try­ing to run full speed down a path that we miss some really impor­tant and fun­da­men­tal facts. Using only cor­rel­a­tive data, we have no way to know the cost to change, the real value that any­thing by itself pro­vides, or the actual scale of impact of any future change.

If effi­ciency is a mea­sure of out­come over cost, then we have no way to have any insight into any piece of that equa­tion. All the analy­sis in the world can not over­come the lim­i­ta­tion of a one direc­tional lim­ited data set from a con­stantly chang­ing and imper­fect ecosys­tem. We find some­thing that sticks out in the data, and then pre­tend that this is the thing that is more valu­able than all the other pieces of data, sim­ply because we can “iden­tify” it, despite the fact that we have no idea of the value of that change nor what some other undis­cov­ered opti­miza­tion would bring. How does know­ing that peo­ple from search spend half as much time as peo­ple come to your site in any way tell you the cost to change their behav­ior? Do not con­fuse your abil­ity to derive value and effi­ciency with your abil­ity to “dis­cover” some­thing in ana­lyt­ics. That you can even change that behav­ior? Or the rel­a­tive scale of impact com­pared to other fea­si­ble alter­na­tives? How is the anom­aly any more effi­cient than the thing that looks like every­thing else? What do you really know from just iden­ti­fy­ing some­thing from cor­rel­a­tive data?

Why do we accept that lim­i­ta­tion and why do we not try to give the con­text nec­es­sary to bet­ter answer those ques­tions? Why do we per­pet­u­ate the myth of only pas­sive data acqui­si­tion as a means to answer so many of the ques­tions that we pre­tend to be able to answer today? Why do we pre­tend we can start with this mag­i­cal data set and some­how arrive at the best answer? We are forced to use con­jec­ture to make assump­tions and then pat our­selves on the back when we get a result. We decide on what we are going to do analy­sis on, find a sin­gle answer, and then defend it because it is backed by data. Is that result a good result? If I have a 100 pos­si­ble pos­i­tive out­comes, and I get the 2nd worst one, who would tell me that is a good thing? Yet when we do not account for that con­text of our answer, we are con­stantly shout­ing our accom­plish­ments from the hill­top. Do we con­grat­u­late the out­come we got or the 98 that we missed? If scored 2% on any test, you would think you failed mis­er­ably, but yet we hide this truth from our­selves to make sure that we all feel like we got an A. The truth is that we will never know any of the impor­tant con­tex­tual infor­ma­tion we seek from cor­rel­a­tive data alone.

Let me pro­pose that test­ing, as a cre­ator of causal data in a con­trolled set­ting is the only way to actu­ally achieve all those value propo­si­tions that you have been promised. That causal data, the seek­ing and cre­ation of it and the use of it as a trans­for­ma­tive agent to power that ear­lier data col­lec­tion, to move past so many of the lim­i­ta­tions of online data col­lec­tion, is the only true way to answer these impor­tant ques­tions. That by “pow­er­ing” your ana­lyt­ics, being will­ing to look past the myths and bad prac­tices, and by break­ing down what you really know from your data is the point where myth become real and where you can truly and dra­mat­i­cally impact your busi­nesses bot­tom line. This is why machine learn­ing is such a big deal, why we move towards opti­miza­tion algo­rithms, and why it is so vital that you under­stand the value propo­si­tions of your var­i­ous types of data. All of those meth­ods lever­age casual infor­ma­tion as a build­ing block to grow and learn. There is a bet­ter way, but it requires you to be hum­ble and dis­ci­plined to reach that “nirvana”.

The core prob­lem with ana­lyt­ics is that you are lim­ited to lin­ear cor­rel­a­tive data. No mat­ter how pretty a model and how much sta­tis­tics you apply, you will never know the value of an action, nor will you know the effi­ciency to change it. We are trapped because the pas­sive nature of the data you are try­ing to use only looks one way (towards the past) and has no way of account­ing for fea­si­ble alter­na­tives, or even the null assump­tion. You are stuck in the land of rates of action; you have 2.8% CTR on your other prod­ucts model on your prod­uct page, but is that good or bad? Even if that is much higher or lower, how do you know that act­ing on it is any bet­ter than act­ing on the thing that looks just like all the oth­ers? If you removed it, where do those clicks go and is that more valu­able than what is there now? Does increas­ing it help or hurt, or more impor­tantly, what hap­pens if it is not there, or what is the cost of chang­ing it as opposed to the cost to change another mod­ule? Are peo­ple who pur­chase more likely to sign up for newslet­ters, or is it the other way around? All of those ques­tions can be answered directly and effi­ciently through test­ing, and once we have cre­ated a num­ber of inter­ac­tions, we can start to see pat­terns from those causal rela­tion­ships. We have the power with very lit­tle effort to start to really see the impact of changes, not just try and extrap­o­late them blindly.

What if you instead ignore all of that data in its pas­sive form, and instead look for the active inter­ac­tion of data to inform those deci­sions? What if instead of start­ing with cor­rel­a­tive data, we ignore it until we have the con­text to make it valu­able? What if we use the causal rela­tions with an eye towards effi­ciency. What if you viewed data as an active mea­sure, one that gains more value the more you elim­i­nate unnec­es­sary waste in the sys­tem, and one that only takes hold once you are dis­ci­plined in how you think about, what you mea­sure, and how you actively change it? What if we stop allow­ing our biases and mis­con­cep­tions of data dic­tate the start of our analy­sis, and instead allow the data to truly tell us what mat­ters? What if you start mea­sur­ing the value of your cor­rel­a­tive data by its inter­ac­tion with the casual data to allow for a much deeper con­nec­tion to effi­ciency. What if you start look­ing for the value of an action, not the rate of an action? Test­ing is your active arm, to change all of that cor­rel­a­tive data into causal data, if you are will­ing to go down that path.

This is the oppo­site of the myth of using ana­lyt­ics to power test­ing, but instead forc­ing your­self to accept that cor­rel­a­tive data, with all the lim­i­ta­tions that are inher­ent in online ana­lyt­ics, is not enough to make mean­ing­ful deci­sions. This is not about using test­ing as a means to prove one point right, but as a means to under­stand and value alter­na­tives against each other. Chang­ing cor­rel­a­tive data into causal data presents you with infor­ma­tion that is truly action­able and that truly gives you insight into the out­comes, value, and costs that we pre­tend we already have the answers for. This is the last step of the evo­lu­tion of look­ing for the best answer and of stop­ping biases from lead­ing you astray.

The chal­lenge is that you can­not just take one test, or any sin­gle data point and pre­tend you have mean­ing­ful infer­ence. Just as you can not pre­tend to know the direc­tion of a cor­re­la­tion or the value of some­thing from its rate of action, you can not just pre­tend to answer every­thing from a sin­gle test result. Div­ing through all that ana­lyt­ics data from a sin­gle test result is a dead end that leads to the same prob­lem that plagues most uses of ana­lyt­ics. You have to be dis­ci­plined and can only reach this point after you have run a full series of tests. Think in terms of using this data to increase the effi­ciency of the sys­tem. You get real value only when you apply test­ing to power your ana­lyt­ics. We can mea­sure the value of the items on the page, their very exis­tence, and the costs to change them. We can quickly get tests live on mul­ti­ple page types and mea­sure the rel­a­tive value. We can run a series of a tests on a page, and induce changes that allow us to see what seg­ments are exploitable, or even what the influ­ence is of var­i­ous parts of a user expe­ri­ence are to those seg­ments. If we are dis­ci­plined, we learn, and we never stop, then we can induce answers to achieve a pos­i­tive result, while also answer­ing those great unknowns that are ignored by ana­lyt­ics alone.

To make this even bet­ter, the act of acquir­ing the data also comes with the ben­e­fit of mean­ing­ful lift and improve­ment to your busi­ness. There is no zero sum game of only acquir­ing data or of get­ting lift, instead using test­ing to power your ana­lyt­ics allows you to meet the needs of change and growth while giv­ing you all the promised panacea that so many claim ana­lyt­ics is pro­vid­ing by itself. It allows you to truly think in terms of effi­ciency and to be able to know the value of the dif­fer­ent fea­si­ble options before you. It requires you to change com­pletely how you think about ana­lyt­ics, to look at as part of a larger ecosys­tem by which you are inform­ing the data, and then using that data to inform future action. It is not just pre­tend­ing that the data is informed and then blindly using it to pre­scribe action. If you instead act to cre­ate casual infor­ma­tion, use that to fil­ter your cor­rel­a­tive data, and do this with dis­ci­pline, you can actu­ally get those answers that we pre­tend we have today.

The sad truth is that most peo­ple who are in test­ing come from an ana­lyt­ics back­ground. Just as many old school mar­keters strug­gle to stay cur­rent in the face of change, so too do many data “experts” who give new names to the same mis­guided tech­niques. They view every­thing through the ana­lyt­ics lens, and as such this makes them want to try and jus­tify their ana­lyt­ics via test­ing, and to apply the same prob­lem­atic dis­ci­plines to test­ing in order to bring it in line with cur­rent efforts. They so want to jus­tify what they have done that they ignore its fun­da­men­tal weak­ness and try to force new dis­ci­plines to con­form to what they are doing. This leads to an entire mar­ket­place full of peo­ple stuck try­ing to jus­tify their exis­tence, but very few will­ing to chal­lenge its entire value propo­si­tion. I chal­lenge you to avoid that black hole, be will­ing to chal­lenge your own world­view and your own core beliefs about data, and to instead look at how you can best get and acquire mean­ing­ful data and how best to lever­age it out­side of what you are com­fort­able with. Very few peo­ple try and look at test­ing as its own dis­ci­pline, or even bet­ter to see how that dis­ci­pline can impact and change how you view other actions. There is a giant fish­bowl of peo­ple who are in a race to the bot­tom jus­ti­fy­ing and preach­ing ana­lyt­ics as a feed­ing sys­tem for test­ing. I chal­lenge you to be bet­ter than the cur­rent environment.

Let me instead sug­gest that you will only achieve real value if you flip that sys­tem, chal­lenge your­self to think out­side of that box, and to power your ana­lyt­ics via your test­ing. Test­ing is just one skill of many, but it deserves its own place at the table, not one that is a fil­ter by which you jus­tify other actions.

Con­clu­sion –

The goal of these posts is to intro­duce new ways of think­ing and to chal­lenge your cur­rent mind­set. I have shown the evo­lu­tion from the most fun­da­men­tal skill to par­a­digms that chal­lenge your entire data world­view. It is only by chang­ing what we do that we grow, and it is only by chal­leng­ing our own core assump­tions about what works that we are able to really make the dra­matic impact to the bot­tom line that we all claim to want to achieve. You can not just accept that every­thing you hold true today will be the same in the future, nor can you expect to get improve if you refuse to change your own behaviors.

The real­ity is that there is no such thing as “right”; in the entirety of human his­tory we con­tinue to find bet­ter answers to all our ques­tions. What I am propos­ing is allow­ing these new ways of think­ing to inter­act with what you are doing and to see if you can then find a newer “righter” answer that brings your pro­gram to a whole new level. It is only through chang­ing our fun­da­men­tal build­ing blocks of what we do that we achieve the scale and impact that we want to achieve. Change who you are, what you think, and let in other ways of think­ing and try to be bet­ter than the water you are swim­ming in. Be will­ing to leave your cur­rent lake and find the diverse ocean of dis­ci­plines and ideas that are out there, and you will always be grow­ing and get­ting bet­ter at what you do.

To nav­i­gate the entire test­ing series:
Test­ing 101 / Test­ing 202 / Test­ing 303 — Part 1 / Test­ing 303 — Part 2