In my last blog arti­cle, Are You Ready for Your Site Redesign?, I dis­cussed how to pre­pare for a new site redesign. In this arti­cle I’ll share rec­om­men­da­tions for the exe­cu­tion of the test of your redesigned site against your cur­rent site. Typ­i­cally these tests are high-profile, high-visibility, high stress efforts and you’ve got to get it right and quickly report per­for­mance to your lead­er­ship. Here’s how…

Iden­tify Entry for Your AB Test

In the event you are lim­it­ing the redesign to the Home­page this step is easy…your entry page is the Home­page. But you may be com­pletely redesign­ing your entire site and so you must iden­tify the pages that you will allow vis­i­tors to enter the test (and thus redi­rect to the new site).

First things first, go to your Site­Cat­a­lyst or ana­lyt­ics tool and iden­tify your top entry pages. Select the 10–15 pages that make up 85% or more of vis­i­tors’ first touch points. Make a list of these pages and their new site equiv­a­lent. Use a spread­sheet to be clear for your devel­oper. This will be the foun­da­tion of your cam­paign struc­ture. As vis­i­tors come to your site by one of these pages you will redi­rect half of them to the new site. Once they are redi­rected, they will be com­pletely inte­grated into your new site (all links within your new site lead to other new site pages) and no fur­ther redi­rect is required.


Cur­rent Page Cur­rent URL New page New URL
Home­page www​.adobe​.com/​i​n​d​e​x​.​h​tml New Home­page www​.adobe​.com/​i​n​d​e​x​2​.​h​tml
Cat­e­gory Page
www​.adobe​.com/​a​c​r​o​b​a​t​.​h​tml New Acro­bat
Cat­e­gory Page
Ana­lyt­ics Page
www​.adobe​.com/​d​i​g​i​t​a​l​a​n​a​l​y​t​i​c​s​.​h​tml New Dig­i­tal
Ana­lyt­ics Page
Solu­tions Help
www​.adobe​.com/​f​i​n​a​n​c​i​a​l​s​o​l​u​t​i​o​n​s​.​h​tml New Finan­cial
Solu­tions Help

In some instances, a page may not have a new site equiv­a­lent and so will need to map to an appro­pri­ate alternative.


Cur­rent Page Cur­rent URL New page New URL
Test­ing FAQ www​.adobe​.com/​t​e​s​t​i​n​g​f​a​q​.​h​tml Test&Target Overview Page www​.adobe​.com/​t​a​r​g​e​t​.​h​tml

Be sure to request that your devel­oper per­sist through the redi­rect any­thing impor­tant in the URL struc­ture such as mar­ket­ing codes or parameters.


Cur­rent Page Cur­rent URL New page New URL
Home­page www​.adobe​.com/​i​n​d​e​x​.​h​t​m​l​?​m​a​r​c​o​m​m​c​o​d​e​=​1​2​3​&​a​m​p​;​i​n​t​e​r​n​a​l​l​i​n​k​=​abc New Home­page www​.adobe​.com/​i​n​d​e​x​2​.​h​t​m​l​?​m​a​r​c​o​m​m​c​o​d​e​=​1​2​3​&​a​m​p​;​i​n​t​e​r​n​a​l​l​i​n​k​=​abc


Define Suc­cess for Your New Site

Suc­cess Met­rics
The hard­est part of a new site launch is estab­lish­ing the KPIs (key per­for­mance indi­ca­tors) that are impor­tant to deter­min­ing the redesign is a suc­cess. But it is also the most impor­tant step. Fail­ing to estab­lish the 1 or pos­si­bly 2 met­rics that will seal the new site’s fate will result in con­fu­sion upon launch (at best) and data over­load (at worst). Be sure to set expec­ta­tions among stake­hold­ers and lead­er­ship of the end goal for this redesign and reit­er­ate it over and over in your doc­u­men­ta­tion and com­mu­ni­ca­tion sur­round­ing the launch.

Test&Target is best used for the high-level sniff test of whether the new site is increas­ing your bot­tom line or whether you should pull it and reeval­u­ate the new designs. Test&Target should be set to report on the ‘WHAT’ (suc­cess actions) as in ‘what are my vis­i­tors doing in the new site’. Are they reach­ing my Prod­uct Pages? Are they Adding-to-Cart? Are they Con­vert­ing? Are they reach­ing the micro­con­ver­sions that I want them to reach and are they doing it at a higher rate than before?

Most pop­u­lar met­rics used to deter­mine success:

  • Total Con­ver­sion Rate, RPV (Rev­enue Per Vis­i­tor), AOV (Aver­age Order Value)= Did your vis­i­tors hit your Thank You page and what was the value of that conversion?
  • Spe­cific Con­ver­sion Rate (such as Approved Appli­ca­tions or Prod­uct X Con­ver­sion Rate)= Were these qual­i­fied con­ver­sions? Did your vis­i­tors con­vert on your most impor­tant products?
  • Add-to-Cart Rate= Did your vis­i­tors declare a preference?
  • Appli­ca­tion Start Rate= Did your vis­i­tors declare a preference?
  • Home­page Click-Through Rate= Did your vis­i­tors engage with the site?
  • Home­page Hero Click-Through Rate= Did your vis­i­tors engage with your most impor­tant promotions?
  • Cus­tomer Sign-in Rate= Are we affect­ing our oppor­tu­nity to per­son­al­ize the site experience?

Site­Cat­a­lyst can expose the “why” (sup­port­ing click actions) as in ‘what are my vis­i­tors click­ing that explains the change in actions in the new site’. Are they click­ing the prod­uct imagery on the Home­page which accounts for a higher rate of Prod­uct Page Views? Are they click­ing the Home­page Hero at a lower rate which explains the reduced con­ver­sion rate of the July promotion?

Use these tools in com­bi­na­tion to tell the full story of what vis­i­tors are doing and how they are get­ting there. Under­stand­ing the user flow by this method explains exactly which ele­ments in your new site is work­ing and which ele­ments may need iter­a­tive opti­miza­tion in a sec­ond or third phase of testing.

Seg­men­ta­tion Strat­egy
And don’t for­get to seg­ment! What are your most impor­tant audi­ences? Watch each behav­ior to ensure you haven’t added any fric­tion in the user flow to one or more groups.

Com­mon seg­ments are:

  • Vis­i­tor type: Cus­tomers, Prospects
  • Cam­paign Refer­rer: Dis­play , email, Paid Search
  • User Device: Desk­top, Mobile, Tablet
  • Browser: Fire­fox, Chrome, Inter­net Explorer
  • CRM data: DINK, Loyal Cus­tomer, At-Risk Cus­tomer, Upgrade eligible

Under­stand­ing how your most impor­tant seg­ments are per­form­ing will direct you to your next iter­a­tions or sub­se­quent tar­get­ing opportunities.

Test Length
Estab­lish a min­i­mum length of time you will run your AB test of the new site. It is often very tempt­ing to call a win­ner as soon as you see favor­able per­for­mance but resist the urge! Test for at least one week to gather a full cycle of vis­i­tors’ activ­i­ties. You may see very dif­fer­ent behav­ior or traf­fic lev­els on week­ends from weekdays.

In addi­tion, watch the cumu­la­tive graph of con­ver­sion rate to look for plateaus in per­for­mance which could indi­cate that the data has sta­bi­lized. If your graph still looks erratic or con­tin­ues to criss-cross with the other expe­ri­ences you need to con­tinue to run the test.

If a test coin­cides with a hol­i­day or pro­mo­tional period, unusual vis­i­tor behav­ior may require that you extend the test to account for atyp­i­cal results. You may even push to delay your launch to avoid such risks to your data.

Estab­lish a Clear Launch Plan

Ensure that your project man­ager has answered at a min­i­mum the fol­low­ing questions:

  • Has the new site been thor­oughly QAed?
  • When will the new site be avail­able in production?
  • When will the Test&Target cam­paign launch? Is it after the new site has been con­firmed in production?
  • If the new site needs to be taken down, who can shut off the Test&Target cam­paign that is direct­ing this traffic?
  • What report­ing is required? Who can pro­vide a 24 and 48 hour check-in of performance?
  • Has the the post-launch results review meet­ing been set?
  • What mar­ket­ing and tech­ni­cal resources are avail­able within the first 4 hours of launch for ques­tions and to mon­i­tor performance?

Deter­mine who owns the test data and who will be inter­pret­ing and com­mu­ni­cat­ing the per­for­mance to lead­er­ship and the rest of the team. Make sure you doc­u­ment the details of the test and the per­for­mance. In your doc­u­men­ta­tion, clearly define each met­ric and segment.

Con­ver­sion Rate is defined as Approved, Pended, or Declined Appli­ca­tions Sub­mit­ted.
Pho­to­shop Con­ver­sion Rate is defined by the Thank You page fir­ing for Pro­duc­tID 123 or 234.

Cre­ate a clear tem­plate to doc­u­ment the per­for­mance of the new site. A proac­tive one-slide sum­mary that includes your hypoth­e­sis, site screen­shots, per­for­mance, and rec­om­men­da­tions (con­tinue to run, push new site, redesign sign-in area) will com­mu­ni­cate per­for­mance at-a-glance and save your inbox and voice­mail from being overloaded.

Iden­tify Next Steps

From your data analy­sis, deter­mine if you can imme­di­ately push your new site to 100% of traf­fic or if first some tweaks are needed. You may find that your site is increas­ing con­ver­sion rate and reduc­ing sign-in rate. Does mean you’ll need to tweak some of the design or will you push to all vis­i­tors and con­tinue to test to opti­mize for sign-ins? Con­firm that all ana­lyt­ics and ven­dor pix­els been prop­erly defined and per­sisted. Are dynamic feeds work­ing as expected or does new code need to be implemented?

Dis­cuss test­ing the addi­tion of suc­cess­fully influ­en­tial ele­ments from the pre­vi­ous site to the new site in the imme­di­ate weeks that fol­low. Estab­lish a Roadmap of iter­a­tive tests that will fol­low your new site push based on any sec­ondary met­rics that fall short.

Enjoy Your Success

New site launches are really hard work. You’ve prob­a­bly worked late, checked data early, and con­firmed and recon­firmed a thou­sand steps dur­ing the launch process. You’ve worked with tons of folks from dif­fer­ent BUs and per­haps sev­eral ven­dors. It’s now time to smile and enjoy the fruits of your hard work.

Test­ing is the only way to pre­dict the new site’s impact to the busi­ness. Con­grat­u­la­tions, you’ve been an inte­gral part of pro­vid­ing real value to your com­pany! You’ve either val­i­dated the invest­ment in a new dig­i­tal expe­ri­ence or you’ve helped avoid a huge mis­take. Either way you are part of set­ting the direc­tion of your company’s online chan­nel. Good job and get mov­ing on your next test!

Test­ing is valu­able as well as fun. Happy testing!


About the author: Christi Ter­je­sen is an Opti­miza­tion Man­ager with Adobe Dig­i­tal, a full ser­vice web opti­miza­tion team within Adobe Con­sult­ing. She helps clients in all ver­ti­cals increase con­ver­sion rate and pro­duce ROI through their test­ing pro­grams. She is a cer­tifed Adobe Test&Target expert based out of New York City.