Build­ing a true opti­miza­tion pro­gram in your com­pany can be a daunt­ing expe­ri­ence. No mat­ter how much you might want to make things work per­fectly, the new­ness of the con­cepts pre­sented, the pol­i­tics around who is “right”, and a hun­dred other fac­tors con­spire against you. Most peo­ple speak about want­ing to get good results, but are often unwill­ing or inca­pable of chang­ing their own behav­iors, let alone oth­ers, in order to get those results. Even worse, there are very few peo­ple who have actu­ally built a world class pro­gram, and they are drowned out in a sea of “experts” who have the one thing you need to do to suc­ceed. With all that infor­ma­tion, where you are men­tally about your pro­gram speaks vol­umes about the value you are get­ting and what the next steps are to really become world class.

No pro­gram is per­fect day one, and almost all of them have to go through some very dif­fi­cult grow­ing pains before they are even func­tional. It is true that every pro­gram fol­lows a sim­i­lar pat­tern of evo­lu­tion, but all pro­grams risk even­tu­ally stop­ping their evo­lu­tion due to a lack of will or under­stand­ing. The chal­lenge starts with the men­tal evo­lu­tion of the pro­gram, since the func­tional parts are mere reflec­tions of where peo­ple view test­ing. It is impor­tant that you under­stand where you are, and where you need to get to in order to succeed.

The chal­lenge is that all pro­grams reach a stop­ping point, either through men­tal exhaus­tion, polit­i­cal push­back, per­sonal ego, or a hun­dred other rea­sons. The key to becom­ing a top pro­gram is to get past that point and con­tinue down the path, even when it seems daunt­ing or does not seem to help you advance politically.

The men­tal evo­lu­tion of programs:

Ran­dom Testing –

All groups start here, think­ing of test­ing as a one off action you take to fig­ure out which piece of cre­ative to show, or which land­ing page is best. No pro­gram is able to achieve the effi­ciency that is nec­es­sary for their pro­gram when they are stuck at this phase, yet most con­ver­sa­tions around test­ing and a great many pro­grams never get past this point. One of the key rea­sons for this is the com­fort and the easy to grasp nature of this stage. This is what most peo­ple think of when they think test­ing, and that is a shame since so many will never see the power it can truly bring to their organization.

The key signs of this stage are: “bet­ter” test­ing, each test is an indi­vid­ual project, you need to get approval for each test con­cept, you have no rules of action. Fun­da­men­tally you are focused on find­ing out who or what is “right”. Test­ing is a one off project that you do when you need to make a deci­sion. Put suc­cinctly, if you are talk­ing about what you want to test, instead of let­ting results tell you what to test, then you know you have not moved past this phase.

One of the other major signs of this stage is the lack of align­ing on site goals. If you have not got­ten align­ment on a sin­gle suc­cess met­ric for all tests, then I can guar­an­tee you are at this point. If you are stuck think­ing about spe­cific met­rics for a test, or think that you will decide how to act and what is impor­tant when you get the results, then you are firmly stuck in this phase. If you try to think or act on the data from tests the same way you do the data from ana­lyt­ics, you can not ever move past this point.

It is pos­si­ble to get value at this stage, and the sad real­ity is that most groups never leave this stage, but ulti­mately you will never have a real opti­miza­tion pro­gram and will be get­ting pen­nies on the dol­lar return if this is where you keep the con­ver­sa­tion. Groups that are stuck in this way of think­ing often think that more tests means more value, and that is true if you are leav­ing the out­come of a test to ran­dom chance. Groups that want to be effi­cient and to get real value from their pro­gram how­ever need to apply those resources not towards run­ning hun­dreds of tests, but instead towards shift­ing the mind­set of the pro­gram to insure higher returns and more long term value from each and every action.

Long Term Site Integration –

The next evo­lu­tion is to start tying in test­ing into larger projects. Work­ing on a redesign? You start test­ing out smaller por­tions on the way. Focused on per­son­al­iza­tion? Then you start test­ing out dif­fer­ent pieces of con­tent or you start test­ing for dif­fer­ent seg­ments. Test­ing has shifted from being a ran­dom action of choos­ing between to choices to one that can shift and change the entire direc­tion or path of a project. To reach this stage, you must be will­ing to shift some part of the project away from what you want to do, and instead choose to do the things that the data tells you to do.

The ben­e­fits of this stage are the start of orga­ni­za­tional build­ing blocks that are fun­da­men­tal to a suc­cess­ful pro­gram. You will have to have agreed on what is suc­cess, you are start­ing to look at test­ing as an effi­ciency tool, and you will have built out some processes to make test­ing more effi­cient. Most likely you have some more ded­i­cated resources and have test­ing as an ongo­ing thing, one that is not just a nov­elty one off. You are stop­ping test­ing what you want, and are let­ting some results from the test deter­mine the path of a project or initiative.

The lim­i­ta­tions are that you are just doing more of the same. You have not really built out a full pro­gram and you are still focused on “bet­ter” ideas. You are just cre­at­ing more struc­ture for the ran­dom­ness of the pre­vi­ous phase, and it is start­ing to shape a direc­tion, but you have not bought in to test­ing as a means to the ques­tion instead of just a way to find an answer. All groups have to get through this stage, and the ones stuck here are going to get more value from the ran­dom test­ing of the ear­lier phase, but it is impor­tant that you focus on mov­ing to the next stage, which is the largest diver­gence on the list.

Dis­ci­plined Base Testing –

This is the real lit­mus test of pro­grams and the largest gain and diver­gence point. Very few groups make this leap, but the ones that do see test­ing as a very dif­fer­ent and more valu­able com­po­nent of their entire orga­ni­za­tion. The keys to this phase are a move­ment away from “bet­ter” test­ing and a change to open ended looks for the most effi­cient ways to apply resources. You are no longer look­ing to test out what you want, but instead using test­ing to under­stand the value of dif­fer­ent alter­na­tives and let­ting the results dic­tate the path of your tests and your ini­tia­tives. All tests do not pre­dis­pose an out­come, instead focus­ing resources con­stantly towards the most effi­cient answer from the prior actions. Test ideas and appeas­ing CXOs becomes sec­ondary to the dis­cov­ery process and the open­ing up of test­ing to dic­tate its own path.

A sign that you have reached this stage is that you no longer look at just a test result, but instead focus on the value of out­comes rel­a­tive to each other. You mea­sure out­comes by the value of rel­a­tive actions and not just that you went from 1 point to another. If you are not prov­ing your­self and oth­ers wrong con­stantly, and if you are not hum­bled by the fact of how lit­tle you really know, then you have not reached this point.

The ben­e­fits are a con­stantly grow­ing under­stand­ing of your site and users, and a move to ensure that all efforts are focused on the most influ­en­tial sec­tions and in the most effi­cient man­ner. You are no longer wor­ry­ing about a “roadmap” or about what won, but about the process of fig­ur­ing it out and con­stantly act­ing. You are start­ing to build out the trust that the sys­tem is only as good as the input, and to not worry about who is “right” as opposed to pro­vid­ing qual­ity dif­fer­en­ti­ated alter­na­tives. You are learn­ing with every action, and you are con­stantly stop­ping cur­rent paths that your orga­ni­za­tion is on and the causal data is dis­cov­er­ing the value of new paths that you never thought of or would not have nor­mally pursued.

The lim­i­ta­tions of this stage is that you are going to upset a lot of peo­ple. You are going to be con­stantly prov­ing that what peo­ple have held dear and believed as the core to their ben­e­fit to an orga­ni­za­tion is actu­ally neg­a­tive. You are going to show that myths passed down for years from schools, experts, and the very thing that CXO peo­ple hold dear is wrong. If you have not built out a cul­ture where you are focus­ing on being wrong, where the goal is actual suc­cess and not prop­a­gat­ing some­ones agenda, you will be deal­ing with con­stant headaches and inter­nal strife. It takes a spe­cial type of per­son to stand up to pres­sure and to do what is right, even if it is not always in their best inter­est. If you are not will­ing to pur­sue results over “glory” then you will most likely not ever reach this phase in your pro­grams growth.

If you have dealt with those issues, even if with just one or two groups, you will see dra­matic improve­ments to the effi­ciency and return on your efforts.

Con­stant Iter­a­tive Testing –

There are very few orga­ni­za­tions that have reached this point. At this phase, all parts of the site are open and con­stantly evolv­ing using dis­ci­pline based test­ing meth­ods in order to grow and to get more effi­cient. There is no longer a view of a project at all, but a con­stant use of mul­ti­ple resources to have a shift­ing and evolv­ing user expe­ri­ence. All jobs in the orga­ni­za­tion have opti­miza­tion as part of their duties, and no longer are you hav­ing debates about what you should test and what you think is bet­ter. Every­one is aligned on a com­mon goal of growth and of prov­ing each other wrong, not right. It is not about the test prov­ing any­thing right, but instead about the qual­ity of the input that is used to feed the sys­tem, which is start­ing to dic­tate just about every part of your user expe­ri­ence and inter­nal resources and initiatives.

Get­ting a pro­gram to build out the mind­set that allows for this phase usu­ally means that you have dealt with all of the neg­a­tives that might arise, and have peo­ple aware of the ben­e­fit of being “wrong”. There is noth­ing that will show the inef­fi­cien­cies of your orga­ni­za­tion faster then try­ing to con­stantly do things that might “hurt” some­one. If you are not will­ing, capa­ble to deal with the pre­vi­ous phases, most likely you will not come close to this level in your program.

Opti­miza­tion Organization –

I refer to this as the myth­i­cal uni­corn, as there are no orga­ni­za­tions in the world that have reached this “nir­vana”. At this point, all con­cepts feed the sys­tem and the sys­tem dic­tates the out­come. This is about let­ting go of wor­ry­ing about who is right, and instead under­stand­ing that you still have to feed the sys­tem, but mak­ing the final deci­sion has to be left to a dis­ci­plined (non-biased and pre­de­ter­mined) use of data, espe­cially causal data, to make those decisions.

No group has reached this point because all orga­ni­za­tions are run by peo­ple, all of which have their own agen­das and all of which need to prove them­selves “supe­rior”. I don’t expect there to ever be a point where peo­ple are capa­ble of putting their egos and pol­i­tics at check, but work­ing towards this point is the only way to really make a true dif­fer­ence and not just use data to push an agenda or to pro­mote yourself.

The evo­lu­tion of pro­grams is a tricky thing, and not one that is quite as black and white as this path might make it seem. What is impor­tant is that you under­stand that you have to shift how you think, and be will­ing to change all the pieces that fol­low, in order to be suc­cess­ful. If you are still think­ing about test­ing as a means of choos­ing between two items, or if you haven’t built out a cul­ture where being wrong is more impor­tant then being right, then all the resources in the world wont make you suc­cess­ful. Suc­cess is not a ran­dom thing, and it is not dic­tated by how many resources you have, but instead by your will­ing­ness to prove your­self and oth­ers wrong. Build­ing out the right mind­set deter­mines the long term value you receive, yet very few take the time to really under­stand or edu­cate oth­ers. There are plenty of mate­r­ial out there that is happy to make you feel good about what­ever stage you find your­self at, but if you really want to get value and really make a suc­cess­ful pro­gram, noth­ing will top con­stant hard work and the will­ing­ness to chal­lenge the norm and to do things against “best practices”.