There is a lot of mis­un­der­stand­ing on the dif­fi­culty of using Adobe Test&Target. There are all sorts of uses of the tool, and in a lot of cases, peo­ple con­fuse their inabil­ity to set-up a proper infra­struc­ture with lim­i­ta­tions of the tool. The won­der­ful thing about the tool is that it has 5000 dif­fer­ent uses. The dif­fi­cult thing about the tool is that it has 5000 dif­fer­ent uses. Fig­ur­ing out what to do, when to use it, and how to tackle that prob­lem is para­mount to a mature program.

Before we get to spe­cific sug­ges­tion how­ever, I want to make it clear what I con­sider a proper tech­ni­cal infra­struc­ture. It comes down to how you answer this ques­tion? Can you get 80% of all tests live, with the proper amount of vari­a­tions, live in 30 min or less. That is from con­cept to launch. That num­ber seems crazy to peo­ple at first, but the real­ity is that you can get a test fully live in 5 min if you have every­thing ready to go. That is the sign of a suc­cess­ful infra­struc­ture: speed of exe­cu­tion, under­stand­ing and flex­i­bil­ity. In order to do that, you have to have things set-up in a way that will make you suc­cess­ful. Here are the key things to do and avoid in that quest for speed and exe­cu­tion with­out sac­ri­fic­ing (in fact improv­ing) results.

Do – Make sure that you can test on all major parts of your site today

Make sure that you have mboxes on all key parts of your site. You will most likely want to make sure that you have mul­ti­ple mboxes on those key pages, usu­ally lever­ag­ing a mix­ture of global and wrapped mboxes through­out the page. Global mboxes are ones that do not wrap any con­tent, but exist at the very top of the page to con­trol CSS, redi­rects, and other func­tions. Wrapped mboxes are ones that wrap key ele­ments of the page, with the best prac­tice of never wrap­ping more then 1/3rd of the view­able page with a sin­gle mbox. The key here is to make sure you are not adding and remov­ing mboxes for each test. One of the nice things is that even if you do have those mboxes on the page, you can dis­able them eas­ily in the con­sole so that you are not charged for them.

Don’t – Have more then 5–6 mboxes on any page

Just because you want to pre­pare a page to suc­ceed does not mean­ing going crazy with mboxes. The real­ity is that each request does have a minor impact on site per­for­mance, and any notice­able inter­ac­tion with the page will result in the valid­ity of a test result being highly ques­tion­able. Keep­ing the num­ber to a mean­ing­ful limit stops this from ever being an issue. Set­ting up a page to be able to do 80% of the things you can do is impor­tant, just as pri­or­i­tiz­ing those efforts to lever­age what you can do today to learn about the other 20% that require addi­tional resources. If you can do 80% of pos­si­ble things today, then do those things, and you will find that often the tests you are not sure about are the ones that pro­duce the biggest and most mean­ing­ful winners.

Do – Make sure that you are track­ing key infor­ma­tion about your users

This is both an on-page and in con­sole ele­ment and may require a bit more time and dis­ci­pline to think about before you act on any­thing. This means lever­ag­ing one of the most under-utilized but extremely valu­able parts of the tool in the pro­file sys­tem. Make sure that you are pass­ing key infor­ma­tion you might have from your CMS or other sys­tems about your users where nec­es­sary, and also make sure that you are record­ing key pieces of infor­ma­tion in the pro­file sys­tem. The key here is to look into it, but not believe it blindly, to search for value and not pre­de­ter­mine out­comes. Just because you really want to tar­get to a cer­tain mem­ber­ship level does not mean you should ever tar­get to them before you test and mea­sure the value of that change com­pared to other alter­na­tives. Even if you aren’t lever­ag­ing it today, hav­ing this type of infor­ma­tion can be lever­aged in the future, and it allows you to eas­ily analy­sis things today that you are not directly think­ing about.

Don’t – Delay your infra­struc­ture for unnec­es­sary pieces of information

Most groups tackle opti­miza­tion as just the end func­tion of a num­ber of dif­fer­ent inputs. The worst exam­ples of this is when groups go through and inter­view for a list of all pos­si­ble pieces of data a group may want to look at or seg­ment infor­ma­tion by. While it might make your life eas­ier since you don’t have to cor­rect any­one, the real­ity is that you are most likely caus­ing mas­sive dam­age to your pro­gram. It is a good idea to get a read on what peo­ple may want, but the pri­mary direc­tive should be to help groups think and act dif­fer­ently with data. As part of that, you need to ask the fun­da­men­tal ques­tion of “will the data I am col­lect­ing improve our busi­ness enough to be worth it, and even so, can I get bet­ter out­comes from using those same resources?” There will be cases when the answer is yes, but in most cases if you are hon­est the answer will be a resound­ing no. In those cases, do not delay other actions just to make oth­ers happy. Use what you have to get the most with what you got, don’t waste time wait­ing for every­thing to be perfect.

Do – Make sure that you can get tests live with­out IT involvement.

You should be able to do 80% of tests with some very basic under­stand­ing of how a web­page works. After you get your infra­struc­ture in place, you do not need IT sup­port except in one off cases. You should not need IT except I extreme cir­cum­stances and only then once you have proven via dis­cov­ery that the item you are using those resources on is the most effi­cient use of everyone’s time. Where groups run into major prob­lems is when they view each test as a tech­ni­cal project, which gives both the wrong impres­sion about the dif­fi­culty of test­ing and makes things much slower to react then they should be. Even worse are groups that think that tech­ni­cally com­plex means valu­able, which is almost always the oppo­site of the truth. Dif­fi­cult means dif­fi­cult, valu­able means valu­able, those two things do not have much to do with the other.

Pri­or­i­tize tests by how much effort they are to get live, as well as how many dif­fer­ent fea­si­ble alter­na­tives you will get, and how much they chal­lenge assump­tions. Do that, and you will dra­mat­i­cally improve the out­comes and reduce the resources needed for your program.

Don’t – QA test­ing like you do every­thing else on your site

There is noth­ing worse for test­ing then hav­ing a test ready to go and then hav­ing to wait weeks for it to go through a stan­dard QA process. Make sure you do a good job of QA, but keep in mind that noth­ing will ever be per­fect, and that you can get most QA done for most tests by sim­ply pass­ing around a few QA links to a cou­ple of col­leagues and hav­ing every­one use a cou­ple of browsers. You will most likely want to do more QA for efforts that dra­mat­i­cally change site func­tion, but the real­ity is that you should be fewer of those tests and far more of the more basic tests any­ways. Add bet­ter rights con­trol so that fewer peo­ple can then push things fully live also adds mean­ing­ful lim­its to make sure you are accom­plish­ing what is needed with­out sac­ri­fic­ing speed and efficiency.

The promise of the tool is that mar­keters start hav­ing con­trol of their site, so why then do peo­ple so will­ingly give up that con­trol? You need to be able to under­stand why you need to QA dif­fer­ently, and how dif­fer­ent types of tests impact site func­tion, so that you can have a mean­ing­ful con­ver­sa­tion with yoru tech team and help them under­stand that you are not going to be con­stantly break­ing their site. Be thor­ough, but don’t go crazy or just dump the respon­si­bil­ity onto others.

Do – Make sure that every­one is aware of when you are run­ning a test

One of the ben­e­fits of set­ting up your cam­paigns to use QA para­me­ters is that those links can be shared with every­one and you can there­for make more peo­ple aware of a cam­paign. There is noth­ing worse than hav­ing a cam­paign live for a few days and then hav­ing to stop it because some VP ran­domly hit a test vari­ant and thinks the site is bro­ken. Com­mu­ni­cat­ing both launch and results and espe­cially lessons learned across tests is also part of a suc­cess­ful infrastructure.

This will also help build aware­ness and inter­est in the results of the tests. If you design tests with enough vari­ants and chal­lenge enough assump­tions, in almost all cases you will get results that make other groups fun­da­men­tally chal­lenge their own ideas of what works.

Don’t – Ever launch a test with­out a clear idea of how you are going to act on the data

There is no point in any test if you are not clear on how you are going to act. This means know­ing your sin­gle suc­cess met­ric, but also what hap­pens to push a win­ner, or how you are going to follow-up, or what you will do with seg­ment infor­ma­tion. Hav­ing those con­ver­sa­tions out­side of the specifics of a test and hav­ing groups aware of their roles before the launch is vital to a speedy and suc­cess­ful infrastructure.

Being pre­pared to act and hav­ing your groups ready to act quickly and effi­ciently can many times be against the entire his­tory of some orga­ni­za­tions. This is why it becomes so vital to build a proper infra­struc­ture on every level to enable test­ing to work. Focus­ing on what mat­ters, instead of just how to get a sin­gle test live, is one of the key dif­fer­ences between groups who just test and those that run suc­cess­ful test­ing pro­grams. Make sure that you focus on the things out­side of a spe­cific test and help move the var­i­ous groups involved towards an end goal, and you will be amazed at how fast you will act and the results you will achieve.

0 comments