One of the most com­mon refrains from estab­lished test­ing orga­ni­za­tions is the need for process. They use the “need” of improved process as the excuse for why they don’t do more, why it takes so long to run tests, why they only get ok results from the pro­gram, and why they don’t impact the busi­ness in a big­ger way. They love to talk about how many tests they can run, but are hes­i­tant to talk about the real impact of the pro­gram. Process becomes this con­stant refrain which serves no pur­pose but to explain that per­sons exis­tence and to insure that the no one gets upset. It doesn’t mat­ter what is going on as long as con­sis­tent action is hap­pen­ing and other peo­ple are using it, as if just the act itself is the provider of value. Process gets turned into this myth­i­cal jab­ber­wocky whose func­tion is sim­ply to make it impos­si­ble for pro­grams to rule the world. Why do groups so eas­ily accept process as the cure all for what ails them, instead of focus­ing on improv­ing the effi­ciency of their program?

Process is just a pat­tern of actions. It’s what those actions are, and how they are lever­aged that truly mat­ter. A great process makes it easy to do the right things quickly and con­sis­tently, but it also enables you do the wrong things con­sis­tently as well. So many groups are lim­ited because they get to a point where they are pro­fi­cient at run­ning a test, but have not spent the time to learn and accept what makes a real method­ol­ogy or strat­egy to test­ing. They think­ing in terms of how to make their lives eas­ier or how do I do more, not about get­ting more from what they are already doing. They accept the first answer they find and are more then happy to prop­a­gate a myth sim­ply because it gets peo­ple to do more. Peo­ple are so happy to get a result from their test that is action­able, that they try to recre­ate ways to do that more and more, instead of focus­ing on ways to get bet­ter results or to increase the like­li­hood of a pos­i­tive out­come. Groups in that stage are forced to try and run more and more tests in order to increase the value of the pro­gram, con­fus­ing excess resource usage with value.

It might seem frus­trat­ing because you want to do more, but the ques­tion is never how do you get more resources or how do you run more tests, but how do you get more from what you do have. Run­ning a test when you start seems like a Sisy­phu­sian effort, but the ques­tion should not be how do you make the boul­der smaller, but what is wait­ing for you when you make it to the top of the hill. It is not ran­dom, nor is it impos­si­ble to get great results con­sis­tently, but you have to change how you think about the world before you can change the hill you are try­ing to climb. You have to stop try­ing to impress or make peo­ple happy and instead truly focus on the dis­ci­pline of mak­ing each action more valu­able then the last. To top it off, there are a large num­ber of groups who con­vince them­selves that the larger hill must have a bet­ter reward, when often the inverse is cor­rect. Climb the right hill, and the path is shorter and the boul­der smaller. Climb the wrong hill, and all you are look­ing for is a smaller boul­der to get off your shoulders.

This is not to say that a proper process can­not dra­mat­i­cally improve a pro­gram, how­ever process becomes a prob­lem when held up as a holy grail as opposed to just a means to an end. I want to walk you through a men­tal exer­cise where we eval­u­ate four pos­si­ble sce­nar­ios for an orga­ni­za­tion, and see what the out­come would be. In this case, we are sim­ply divid­ing groups who have proper process in place and those that have proper dis­ci­pline and method­ol­ogy in place.

Group #1: Poor Process & Poor Strategy

Groups in this quad­rant suf­fer from resource shock for test­ing. They have accepted that they might only see results any­thing pos­i­tive from 25–50% of their tests and are strug­gling to run a num­ber of tests. They are often frus­trated by their inabil­ity to do more, and as such they try to make up for it by mak­ing larger and more com­plex tests, often requir­ing even more resources at each turn and slow­ing down the process even more. They are often run­ning tests in-line with other groups in the com­pany and try­ing to pig­gy­back their efforts to show that they add value.

Most orga­ni­za­tion start here and sadly never leave, even when they dump resources into hir­ing more peo­ple or hir­ing an agency. Groups in this phase suf­fer from incon­sis­tent returns, low value, and poor adop­tion in the orga­ni­za­tion. The only sav­ing grace is that they are saved by the low amount of resources that are dumped into the pro­gram and by the small expo­sure high­light­ing their inef­fi­ciency. To make it worse, agen­cies love to keep peo­ple in this state, as it makes their efforts to get a result seem much greater then they really are and it jus­ti­fies mas­sive amounts of hours to achieve these minus­cule results. They get one or two results over long peri­ods of time and then spend hours jus­ti­fy­ing it as a great exam­ple of the power of their efforts.

At this point, you are a weak man try­ing to push a rock up a large hill that you are not sure what is on the other side of. It can be done, but it takes a long time to succeed.

Group #2: Poor Process & Proper Strategy

Orga­ni­za­tions in this phase have very poor resources, low com­mu­ni­ca­tion with other groups, and only the most basic of infra­struc­tures on the site. They have to fight IT and prod­uct man­age­ment and are often stuck run­ning a few tests a month or quar­ter. The trade­off is that they are max­i­miz­ing their return through proper use of resources, learn­ing, and best not bet­ter test­ing. They are at a state where they have lots of road­blocks and low resources avail­able to them, but they are plan­ning their pro­gram around the most effi­cient ways to exploit the resources they. They are not set­tling for the excuses, nor are they run­ning a per­fect pro­gram, but they are focus­ing the test­ing pro­gram now that they have instead of try­ing to make what lit­tle they have fit into a pre­de­ter­mined mind­set of what “suc­cess” is. They are focus­ing on using what they have to its max­i­mum return instead of facil­i­tat­ing oth­ers ideas or “test ideas”. They are not try­ing to fit a square peg into a round hole, but instead try­ing to find what fits best into the hole they have access to.

At this stage, you expect that every test they run will give them mul­ti­ple pieces of infor­ma­tion about the value of fea­si­ble alter­na­tives against each other and expect that 100% of tests will feed the next test and pro­duce action­able insight. They expect that 75%-95% of tests will also deliver action­able mean­ing­ful lift of mul­ti­ple alter­na­tives which they can then use to mea­sure the effi­ciency against each other to choose the best option. They might only run a few tests a month, but they are usu­ally get­ting mag­ni­tudes greater direct return from their entire pro­gram then pro­grams spend­ing 20x on resources.

Groups in this phase use very lit­tle resources and due to that are not able to move at the speed they want. These groups though are able to max­i­mize their effi­ciency by focus­ing on get­ting the largest return for what they have and mak­ing sure that every action returns pos­i­tive and mean­ing­ful infor­ma­tion. They could do more with bet­ter processes and greater resources, but they are still pro­duc­ing results that knock other ini­tia­tives out of the water.

At this point, you are still a weak man, but you have cho­sen shorter hills and you have a really good idea what is wait­ing for you on the other side. You are not let­ting some per­son miles away dic­tate the hill your go over, but instead find­ing the best use of resources. Each hill you peak points you to the next hill to tackle instead of pre­dis­pos­ing the order of hills that you will tackle.

Group #3: Good Process & Poor Strategy

Groups in this phase have accepted that test­ing is a great thing for their orga­ni­za­tion. They love to test and are great at get­ting tests live. They can often expect to get hun­dreds of tests live a year and are great at assign­ing resources, cre­at­ing a char­ter, hav­ing QA go through a test, and have mul­ti­ple resources assigned to get a test live and going. Often times they might also work with agen­cies and pay out­side peo­ple to make sure that they have those resources. They are still only get­ting results from 25–50% of their tests, and are often find­ing that they have to spend more and more resources to get larger and big­ger cre­ative as they run through their “roadmap”, but at least they do have results because they have run so many tests. Often times groups in this stage have come up with non mean­ing­ful mea­sures of suc­cess, such as click through rate or bounce rate as a mea­sure in order to prove to exec­u­tives that they are doing far more then they really are. This is the point that a large num­ber of very mature pro­grams find themselves.

Groups in this phase have large dol­lar fig­ures they can point to for out­comes, but when looked at for effi­ciency are often get­ting very poor ROI com­pared to effi­cient test­ing pro­grams and are tak­ing resources away from other efforts in the orga­ni­za­tion. They have built an empire for the peo­ple run­ning the pro­gram and because of the nature of test­ing, are often one of the top rev­enue dri­vers for the com­pany, but are dump­ing money to do so. I love that these groups appre­ci­ate and believe in test­ing to the degree that they do, but in many ways not max­i­miz­ing their invest­ments. By prop­a­gat­ing the myth that more tests equals bet­ter results, they allow peo­ple to con­fuse get­ting a result from a test with get­ting a mean­ing­ful result of viable alter­na­tives. Often times these groups get about the same mon­e­tary return from their pro­grams as the orga­ni­za­tions in group #2 above, but are spend­ing mag­ni­tudes more on resources to do so.

At this point, you are spend­ing money for a giant crew and large indus­trial equip­ment to push the same rock up usu­ally now steeper hills where you still don’t know what is wait­ing for you on the other side. There are lots of groups out there wait­ing and ask­ing to take your money to be the crew and to give you the tools so that you too can mount this hill.

Group #4: Good Process & Proper Strategy

There are very few groups that are in this quad­rant, but those that con­stantly see returns and have built out their pro­grams so that they can test every­thing and have every­one on the same page to act and test what is nec­es­sary. Because they are being effi­cient and learn­ing as they go, each test expo­nen­tially has a higher chance of a pos­i­tive return and they are hit­ting on 90–100% of their tests to gen­er­ate at least 2–5% lift and to com­pare alter­na­tives. They will run a num­ber of tests, but they focus on the out­comes of the tests and not the raw num­ber it takes to get some­where. They are also chang­ing the path of the entire orga­ni­za­tion, show­ing the causal value of dif­fer­ent alter­na­tives and help­ing to stop cur­rent activ­i­ties while dis­cover new ones.

Groups in this phase see mag­ni­tudes higher gross return on their efforts. They are often not spend­ing much more in the way of resources then orga­ni­za­tions in group #1, and they are mak­ing high mul­ti­ples of the gross returns of groups in group #3.

At this point, you have got­ten some peo­ple that know how to move a boul­der, and you have bought some equip­ment to move it, but you are cho­sen much bet­ter hills with much more con­sis­tent returns and you are choos­ing to point the peo­ple only at the hills that have the high­est returns. The boul­der has become irrel­e­vant to the hills you are tackling.

A proper process and more resources sim­ply high­light the dimen­sions of your pro­gram. If you are inef­fi­cient or do not know how to run a proper pro­gram, it allows you to do more and get a higher gross return. There are many peo­ple who hide behind process in order to jus­tify their jobs, and that is a shame, as they could do so much more if they are just will­ing to play less pol­i­tics and focus more on achiev­ing results. If you care about being effi­cient and about get­ting the max­i­mum amount of return how­ever, then you have to focus on both parts, and under­stand that more is not always bet­ter. If you are only hit­ting on 50% or less of your tests, or you only get a sin­gle out­come from your tests, then you have to run more than dou­ble the num­ber of tests in order to get the same gross return of some­one who is hit­ting on 100% of them (com­pound returns and effi­ciency of mea­sur­ing mul­ti­ple alter­na­tives against each other).

It might seem frus­trat­ing because you want to do more, but the ques­tion is never how do you get more resources or how do you run more tests, but how do you get more from what you do have. It is easy to blame process or to think that a bet­ter process some­how cre­ates a mag­i­cal panacea that solves every­thing, but that is because we are look­ing exter­nally. It is far eas­ier to blame oth­ers then to look inter­nally and make sure you are doing the most with what you have. Be hon­est with your­self and use what you have, and you will be amazed at the results you achieve and how lit­tle in the way of resources you need to achieve them. Change how you and oth­ers think first, and then process can truly make a big impact.