As per­son­al­iza­tion and test­ing con­tin­ues to become more and more main­stream you are start­ing to see a whole slew of groups that are being intro­duced to test­ing, or who may believe they have more func­tional opti­miza­tion knowl­edge then real­ity. So many groups would get be bet­ter off if they just avoided a num­ber of com­mon pit­falls that befall new pro­grams. While ear­lier I put together my list of the top 7 deadly sins of test­ing (no sin­gle suc­cess met­ric, weak lead­er­ship, fail­ure to focus on effi­ciency, test­ing only what you want, con­fus­ing rate and value, and falling for the grave­yard of knowl­edge), I want to instead give you a very quick check­list to make sure that at least your first few efforts in test­ing are not more painful and far more fruit­ful then what nor­mally happens.

It does not mat­ter if you have tested before, or are new to test­ing. What mat­ters instead is how you tackle today’s issues and how you set your­self up to suc­ceed. Break­ing down the com­po­nents of a suc­cess­ful first test allows you see the action items, and allows you to move towards the moments that really make or break a pro­gram. Noth­ing makes everyone’s life harder than start­ing out test­ing on the wrong foot, since every­one will think that is how things are going to hap­pen from then on. With that in mind, if you do noth­ing but fol­low this sim­ple check­list, you are going to be far bet­ter off.

1) Decide on what you are try­ing to accom­plish – Pick one met­ric for all your tests, across your entire site or org that defines suc­cess. This might be easy or hard, but it is vital. This is espe­cially dif­fi­cult for peo­ple com­ing from an ana­lyt­ics back­ground who are used to report­ing on mul­ti­ple things and find­ing stories.

2) Pick one or two pages to start – Do not try to eat the entire ele­phant in one sitting.

3) Do not focus on your test ideas – You are going to have test ideas in mind, and you are going to want to only talk about and focus resources on that one test. With­out fail this is where most groups want to focus, but I can­not stress enough how not impor­tant your con­cept for improve­ment will be to a suc­cess­ful program.

4) Make sure your first test is very sim­ple – Don’t start try­ing to do a full redesign of your entire site or com­pletely change a user flow. Pick one page and one thing to start. If you are not sure, then pick a page and design your first test to mea­sure the rel­a­tive value of all the items on the page.

5) Decide on your ini­tial seg­ments – Make sure all tests have a seg­ment list. Not being focus on a spe­cific seg­ment will make it far eas­ier to find exploitable seg­ments and will start the process of learn­ing even if you do not intend to use them right away.

Here are some basic rules for seg­ments to make your life easier:


• Must be at least 5% of your total pop­u­la­tion (10% of a smaller site). This is total, not iden­ti­fied traf­fic
• Must have a com­pa­ra­ble mea­sure (you can’t mea­sure new users ver­sus Google users, since there are new Google users).
• Have to be defined PRIOR to the start of any cam­paign
• Need to cover the 4 big user infor­ma­tion types (the listed items are just examples):

Tem­po­ral
Day part­ing
Day of week

Phys­i­cal
Geo loca­tion
Browser
Oper­at­ing sys­tem (basi­cally all the stuff you get from the user agent string)

Refer­rer
How did the user get to the site
Key word types
Refer­rers
Channels

Behav­ior
New/returning
Used inter­nal search before
Purchaser/non purchaser

6) Start build­ing a proper infra­struc­ture — Build out a tech­ni­cal frame­workfor your entire site for test­ing. This may not be tied to the first test, though that will be part of it. Get access so that you can run tests on 80% of your pages, using a com­bi­na­tion of local and global set-up. A lit­tle pain up front will save you from lots of pain later on. It is always best to avoid death by a thou­sand cuts when­ever pos­si­ble, even if you don’t see that issue immediately.

7) Decide on rules of action – Make sure every­one is very clear on how you are going to call a test and act on win­ners before you launch your first test.

8) Mak­ing sure you are not going to QA tests like you do other parts of your site – So many test­ing pro­grams are destroyed by let­ting IT decide on the work flow and on how you are going to QA your tests. You may have to work with your IT group, but it is vital that test­ing is not owned by your IT group but instead your mar­ket­ing and prod­uct teams.

9) Point your resources towards fea­si­ble alter­na­tives, not adding more met­rics — Any addi­tional time and addi­tional resources need to be used on the fol­low­ing two things:

A) Adding as many alter­na­tives to the test as pos­si­ble, traf­fic per­mit­ting
B) Edu­cat­ing and work­ing with groups to make sure they under­stand why you are test­ing and how you are going to act.

10) Remem­ber that the most impor­tant time for your pro­gram is not the first test – Your first test is fun to focus on, but the period after your first test is where you can start to really focus on test­ing dis­ci­pline. This is the time that defines whether you get good at just run­ning tests, or if you build a rock star opti­miza­tion pro­gram. So many groups fail because they miss how vital this time is.

There you go, 10 sim­ple steps to make sure that your first moments in test­ing do not take your pro­gram astray. This is hardly the end of the road, but if you sim­ply avoid set­ting your­self up for fail­ure, then you can really start to look ahead to all the oppor­tu­nity that is out there.

0 comments