As more and more pro­grams start to emerge with the growth of the online opti­miza­tion field, there becomes a pre­pon­der­ance of “best prac­tices” when it comes to test­ing, per­son­al­iza­tion, and all other active forms of lever­ag­ing data, it seems like you have to know a mas­sive amount to just under­stand com­pletely what those “experts” are saying.

With that in mind, I wanted to present some very sim­ple do’s and don’ts for pro­grams just get­ting going. Start­ing cor­rectly and set­ting the stage for suc­cess is vital to really being effi­cient and get­ting value out of your pro­gram that you can and should. The prob­lem is that in almost all cases people’s first instincts lead them astray. What you don’t do is more impor­tant usu­ally then what you do choose to do. The key is to make sure that you focus your lim­ited time on the actions that will pro­vide the great­est growth and value to your pro­gram. The same advice can work for groups that have been test­ing for years as many of those pro­grams also are just built up ver­sions of the same bad behaviors.

DO – Hold dis­cus­sions about a sin­gle suc­cess metric

The very first and some­times the most painful hur­dle that a pro­gram faces is get­ting groups to agree on how to act. This is in many ways com­pletely counter cul­ture as many groups have com­pet­ing goals and are only focused on their lit­tle piece of the larger pie. If you do noth­ing else, get­ting peo­ple to agree on the one thing that you can all make a deci­sion on is vital.

A side ben­e­fit of this con­ver­sa­tion is that it starts the process of allow­ing peo­ple to dis­so­ci­ate the actions they think will lead to suc­cess and the actual mea­sure of suc­cess. Way too many peo­ple think that if their idea is more peo­ple look­ing at prod­uct X will gen­er­ate addi­tional rev­enue then the mea­sure of suc­cess is more peo­ple look­ing at prod­uct X. You may have an idea for what you want to do, but you are doing it to accom­plish a goal, so mea­sure the goal, not the action. The mea­sure of suc­cess would be addi­tional rev­enue, and once that is the only goal, you can start com­par­ing all fea­si­ble ways to achieve that exact goal.

DON’T – Get too caught up on test ideas

Some of the least impor­tant parts of a test­ing pro­gram is the gen­er­a­tion of test ideas. While this is the fun part for peo­ple to try and prove their point, the keys to suc­cess are not in hav­ing a bunch of ideas, but in putting together the infra­struc­ture and help­ing under­stand the dis­ci­pline of suc­cess­ful test­ing. Test ideas will come nat­u­rally out of every­day con­ver­sa­tions and espe­cially out of prior tests and learned knowl­edge. There is never a lack of things you can do, but focus­ing too much on that part allows peo­ple to get caught up in many dif­fer­ent biases which will make their ratio­nal eval­u­a­tion of the results to col­lide with their ego.

DO – Apply tech resources on a larger infrastructure

All tools require some sort of deploy­ment, and while some are eas­ier than oth­ers, the biggest mis­take you can do is to think that every test will require mas­sive amount of resources. If you build a proper infra­struc­ture across your site, then most tests will not require any involve­ment from devel­op­ment resources whatsoever.

The key to a good infra­struc­ture is to have tag­ging in the key loca­tions on your top pages so that you can test just about any­thing. You will also need to make sure that you have track­ing in place for your suc­cess met­ric, and for any addi­tional infor­ma­tion (like seg­ment infor­ma­tion) that you may want to provide.

Test­ing should not be thought of as a project but as an ongo­ing orga­ni­za­tion and site fea­ture. It is some­thing that should be set-up in a way to never stop and to never be about the sim­ple val­i­da­tion of a sin­gle idea. In order to max­i­mize this, going through the ini­tial “pain” of a larger deploy­ment and mak­ing sure that your IT group under­stands that this is not a per­ma­nent engi­neer­ing owned project will dra­mat­i­cally improve your abil­ity to move quickly later on. The key once this is done is to pri­or­i­tize tests based on resource usage and pri­or­i­tize tests that will deliver the great­est return for the low­est resource usage.

DON’T – Think test­ing is just an exten­sion of your ana­lyt­ics group

How you think about opti­miza­tion is almost the exact oppo­site of ana­lyt­ics. Instead of pat­terns and anom­alies of larger data sets, you have a sin­gle point and the push to make con­sis­tent mean­ing­ful changes. Test­ing is not just the action arm of some analy­sis you did to val­i­date your point, it is the active acqui­si­tion and inter­ac­tion with data.

To suc­ceed, you need to think about seg­men­ta­tion dif­fer­ently. You need to think about what a suc­cess met­ric really is and how it is dif­fer­ent in test­ing. You need to able to speak in terms of com­par­a­tive analy­sis, not val­i­da­tion. Basi­cally, you have to be able to turn just about every­thing you do with ana­lyt­ics on its side. Later on, you can start lever­ag­ing the two together, but as you start, sep­a­rat­ing them com­pletely is going to grant you far more return with far less work then try­ing to just tack test­ing into your ana­lyt­ics daily activities.

DO – Think about your rights management

Make sure you know who is going to have what rights and make sure you have some checks in place from too many peo­ple chang­ing your site.

DON’T – Blindly fol­low sta­tis­ti­cal measures

You don’t need to know every­thing about all sta­tis­tics, but you do need to under­stand some basic con­cepts to really under­stand results. The first is that for any sta­tis­ti­cal tool to be use­ful, you need not just sta­tis­ti­cal con­fi­dence, but you also need the data to be rep­re­sen­ta­tive of the change you are going to make. If you get 99% con­fi­dence in 3 hours on a Fri­day after­noon, that data is only rep­re­sen­ta­tive of that period of Fri­day afternoon.

DO – Start­ing think­ing how you are going to store and share results

When you are test­ing right, you are going to con­stantly learn new things and if you are doing your test­ing right these lessons you learn will even­tu­ally be far more valu­able than any indi­vid­ual result. You need to start think­ing about where you are going to share this infor­ma­tion, the for­mat, and the avail­abil­ity. You also need to make sure that this is not a sta­tic item but a liv­ing knowl­edge base.

DON’T – Let any test go out with just two recipes

One of the hard­est lessons to learn is that test­ing is not about val­i­da­tion of a sin­gle point, but about com­par­ing fea­si­ble alter­na­tives and being pre­pared to go in direc­tions that you never imag­ined. While not every­one will be ready for this day one, the sim­plest way to pre­pare peo­ple is to force dis­ci­pline on them. Mak­ing peo­ple have mul­ti­ple very dif­fer­ent but fea­si­ble alter­na­tives will start giv­ing you far more infor­ma­tion and will start to show you areas where what they thought mat­tered didn’t.

There are a thou­sand other things that go into run­ning a pro­gram, but just start­ing out, if you tackle these sim­ple things and avoid some com­mon traps, and then you will be get far more results, make a larger impact to your busi­ness, and use far less resources. Think about what you really want from your pro­gram and then stop focus­ing on indi­vid­ual tasks and instead start putting the key pieces in place for long term success.