The beauty of any pro­gram or any skill is that no mat­ter where you are, or what you have accom­plished, you can always get bet­ter. There is no such thing as the per­fect test­ing orga­ni­za­tion, or a per­fect way to use data, which means there are always oppor­tu­ni­ties to grow and to get bet­ter. With that in mind, I want to present var­i­ous skills and dis­ci­plines that will allow your pro­gram to grow no mat­ter where you are at. If you have just started out, been test­ing for years, or are a mas­sive opti­miza­tion orga­ni­za­tion, there are always new ways to think and new skills to allow you to get more value out of each and every action that you take.

All pro­grams are built off of key fun­da­men­tal dis­ci­plines that allow all other dis­ci­plines to grow and to be acted on accord­ingly. All groups start out by need­ing to grow past the ini­tial chal­lenges and lim­i­ta­tions that face your pro­gram. Most groups find a point where test­ing has shown a few wins, you find places to test where it makes sense, but you haven’t been able to get the level of con­sis­tency or buy­ing that you are look­ing for. You are trapped in a world of explain­ing why you want to run a test, or work­ing with dif­fer­ent groups who aren’t inter­ested in what the rest of the orga­ni­za­tion is doing. Early devel­op­ment pro­grams have many fac­tors that unite them; you might have been test­ing for 3 days or 3 years, and yet if you haven’t really taken the time to grow and incor­po­rate test­ing beyond a super­fi­cial way, you will never achieve the true value that test­ing can pro­vide for your organization.

The first thing that dif­fer­en­ti­ates pro­grams is the orga­ni­za­tional pieces that have been put in place to help them move at a speed that will truly impact the bot­tom line. With that in mind, here are 5 key dis­ci­plines that will allow your pro­gram to the next level. For any test­ing pro­gram to work long term and to grow, you must first address the orga­ni­za­tional road­blocks that will with­out fail trip up your pro­gram. The first and some­times most vital aspect of grow­ing your pro­gram can be adop­tion and the real­iza­tion that this is not a once in a while type of action that you take.

Sin­gle Suc­cess Metric –

With­out fail, the sin­gle great­est deter­min­ing fac­tor in the over­all suc­cess of your test­ing pro­gram is have you uni­fied on what you are try­ing to achieve and are you opti­miz­ing for the end behav­ior you want. So many pro­grams fail when they try to opti­mize for the next page or a small action, or even worse to try and opti­mize for mul­ti­ple met­rics to make every­one happy. So many pro­grams are less the sum of their parts, and because of this they are never able to impact the fun­da­men­tal bot­tom line of the busi­ness at the level that they are capa­ble of.

It isn’t just about opti­miz­ing for the goal of the site and not focus­ing on a indi­vid­ual con­cept, you must be able to answer all ques­tions of this for­mat: If A goes up, and B goes down, what do we do? What about A and C? Ulti­mately, you will quickly dis­cover there is one thing that you really have to go up to be suc­cess­ful. It isn’t about clicks, our bounce rate, or just mov­ing peo­ple to another part of your site, it is about focus­ing on what all of those behav­iors are try­ing to accom­plish. In just about all cases, the met­ric you will end up using is directly tied to how you make money, some­thing like RPV, or lead con­ver­sion rate, or an engage­ment met­ric. What is impor­tant is that is global across your entire site, not just the sec­tion you think you will impact. That is what you need to focus on. You need to have every­one in agree­ment on how to act on the test before it launches, so that you don’t lose momen­tum or get lost in the inter­nal quick­sand of politics.

Noth­ing allows your pro­gram to be more valu­able and for you to bring peo­ple together then to have every­one pointed towards the same end goal. Noth­ing ends inter­nal debates faster than not wor­ry­ing about clicks or sign-ups or search ver­sus organic then opti­miz­ing for rev­enue and giv­ing all actions an equal footing.

When I come in to assist groups and make sure they are get­ting the most value, noth­ing is more vital and I will refuse to go for­ward unless we can get agree­ment on what defines suc­cess and how we are going to mea­sure it. You are often defined by what you don’t do as much as what you choose to do, and this is the sem­i­nal moment when you can take a giant leap in your pro­gram. This is the moment when a new dis­ci­pline, unique to opti­miza­tion, comes into place. It also allows us to treat any idea demo­c­ra­t­i­cally and let the num­bers define success.

Con­sis­tent Resources –

One of the many fail­ings of early pro­grams it he lack of own­er­ship of the process and tool. Does IT, mar­ket­ing, prod­uct, mer­chan­dis­ing own test­ing? Whose resources get assigned and who will run this spe­cific test? Do you have a FTE, or just a few ran­dom hours assigned to some­one who has 10 other tasks on their plate? It is vital that as you build your pro­gram, that you have con­sis­tent time and resources assigned, out­side of any spe­cific test or project. The skills of test­ing, not just the tool, but under­stand­ing how your site works, how to move things, test­ing dis­ci­plines and how to nav­i­gate inter­nal pol­i­tics, all of these take expe­ri­ence and time. Pro­grams that have new peo­ple jump­ing on board, peo­ple who run a test once or twice a quar­ter, or who have not clearly iden­ti­fied who will set-up, run, and do report­ing on tests have the hard­est time cre­at­ing the momen­tum and cadence that allow your pro­gram to really blos­som and to help insure that you are get­ting the mag­ni­tudes of value that test­ing can provide.

A good first step for any pro­gram is to assign a set num­ber of hours per week. Assume that you will always be test­ing, and such, this won’t be tied to any indi­vid­ual test. As a good next step, you can make sure that any test you run can be done with that amount of resources in one week. Assign­ing the same peo­ple, a set amount of time, and mak­ing sure that you use it to always be mov­ing for­ward will help your pro­gram achieve the cadence and scope of impact that you are look­ing for.

Infra­struc­ture –

Hav­ing a proper infra­struc­ture on your site is vital to be able to have a real pro­gram. The pieces of your infra­struc­ture has two main com­po­nents, the site code and the orga­ni­za­tional com­po­nents. A good robust infra­struc­ture on your site should allow you to get a new test live within a day on any impor­tant part of your site with­out IT involve­ment. If you are cre­at­ing test­ing code for each test, or if you require a sep­a­rate ticket and full QA process for each test, you will never be able to move at the speed that a suc­cess­ful pro­gram runs at.

The other com­po­nent for infra­struc­ture is the orga­ni­za­tional com­po­nents. Do the impor­tant groups in your orga­ni­za­tion under­stand how you will run a test? Why you run a test? What will define suc­cess and how best to think about test­ing to pro­vide mean­ing­ful answers? Are they bought in to test­ing or do they see it as some­thing that threat­ens their immi­nent domain? Whereas the tech­ni­cal infra­struc­ture is some­thing that may be done mostly at the start and then revis­ited from time to time, the edu­cat­ing and work­ing with var­i­ous groups in your com­pany is an ongo­ing thing and is vital for long term growth.

Rules of Action –

Once you have fig­ured out your sin­gle suc­cess met­ric, do you have a con­sis­tent set of rules to define how you call a win­ner? How about when you have a win­ner, how does it go live, who is respon­si­ble, what is the time­frame? How do you build off of a test result? It is vital that you answer these ques­tions and get peo­ple in agree­ment BEFORE you launch a test. Suc­cess and fail­ure of any test is not in the lift it pro­vides, but your abil­ity to act on that data in a clear and expe­dited manner.

Groups will often write these down in a doc­u­ment and get sig­noff from dif­fer­ent leads of the orga­ni­za­tion, or cre­ate a wiki and pro­vide access to every­one. Each group has their own unique ver­sion of this, but it is vital that before you get to the action phase, that every­one is clear on how and why you do what you do. Noth­ing kills a pro­gram faster than peo­ple des­per­ately try­ing to stop a change because they were “wrong” or peo­ple not will­ing to act on data because they were not ready to go. Never start a test if you are not able to act on the results, and rules of action are your way to ensure that you will be able to do this.

Pro­gram Champion –

Out­side of hav­ing a sin­gle suc­cess met­ric, there may be no greater deter­mi­na­tion of long term suc­cess than hav­ing some­one with some power in your orga­ni­za­tion who spon­sors and pushes test­ing. Hav­ing a per­son who can nav­i­gate the polit­i­cal waters to enable the dis­ci­plines listed above, and to help com­mu­ni­cate and hold peo­ple account­able for results. This per­son may not be involved in day to day actions, but they will often hold reg­u­lar review meet­ings, talk to dif­fer­ent groups, and present results to senior lead­er­ship. It is not about “own­ing” test­ing, opti­miza­tion is part of everyone’s job, but this per­son helps get test­ing the atten­tion it needs and holds peo­ple accountable.

You can look at any of the best test­ing orga­ni­za­tions, and with­out fail you can name this per­son instantly and clearly. That per­son has bought in and while the pro­gram is often not per­fect, they have helped lead it in a way that there is no doubt who spon­sors it, why they push it, and how much value it brings to your company.

These dis­ci­plines rep­re­sent just the first steps to really becomes a opti­miza­tion leader, but they are vital in order for you to be able to grow and to facil­i­tate the adop­tion of more advances skills. In the sec­ond set of dis­ci­plines, we will explore ways to think about test­ing to insure that you are get­ting more value from each of the actions which you just helped facilitate.

To nav­i­gate the entire test­ing series:
Test­ing 101 / Test­ing 202 / Test­ing 303 — Part 1 / Test­ing 303 — Part 2