In my last blog article, Are You Ready for Your Site Redesign?, I discussed how to prepare for a new site redesign. In this article I’ll share recommendations for the execution of the test of your redesigned site against your current site. Typically these tests are high-profile, high-visibility, high stress efforts and you’ve got to get it right and quickly report performance to your leadership. Here’s how…

Identify Entry for Your AB Test

In the event you are limiting the redesign to the Homepage this step is easy…your entry page is the Homepage. But you may be completely redesigning your entire site and so you must identify the pages that you will allow visitors to enter the test (and thus redirect to the new site).

First things first, go to your SiteCatalyst or analytics tool and identify your top entry pages. Select the 10-15 pages that make up 85% or more of visitors’ first touch points. Make a list of these pages and their new site equivalent. Use a spreadsheet to be clear for your developer. This will be the foundation of your campaign structure. As visitors come to your site by one of these pages you will redirect half of them to the new site. Once they are redirected, they will be completely integrated into your new site (all links within your new site lead to other new site pages) and no further redirect is required.


Current Page Current URL New page New URL
Homepage New Homepage
Category Page New Acrobat
Category Page
Analytics Page New Digital
Analytics Page
Solutions Help New Financial
Solutions Help

In some instances, a page may not have a new site equivalent and so will need to map to an appropriate alternative.


Current Page Current URL New page New URL
Testing FAQ Test&Target Overview Page

Be sure to request that your developer persist through the redirect anything important in the URL structure such as marketing codes or parameters.


Current Page Current URL New page New URL
Homepage New Homepage


Define Success for Your New Site

Success Metrics
The hardest part of a new site launch is establishing the KPIs (key performance indicators) that are important to determining the redesign is a success. But it is also the most important step. Failing to establish the 1 or possibly 2 metrics that will seal the new site’s fate will result in confusion upon launch (at best) and data overload (at worst). Be sure to set expectations among stakeholders and leadership of the end goal for this redesign and reiterate it over and over in your documentation and communication surrounding the launch.

Test&Target is best used for the high-level sniff test of whether the new site is increasing your bottom line or whether you should pull it and reevaluate the new designs. Test&Target should be set to report on the ‘WHAT’ (success actions) as in ‘what are my visitors doing in the new site’. Are they reaching my Product Pages? Are they Adding-to-Cart? Are they Converting? Are they reaching the microconversions that I want them to reach and are they doing it at a higher rate than before?

Most popular metrics used to determine success:

  • Total Conversion Rate, RPV (Revenue Per Visitor), AOV (Average Order Value)= Did your visitors hit your Thank You page and what was the value of that conversion?
  • Specific Conversion Rate (such as Approved Applications or Product X Conversion Rate)= Were these qualified conversions? Did your visitors convert on your most important products?
  • Add-to-Cart Rate= Did your visitors declare a preference?
  • Application Start Rate= Did your visitors declare a preference?
  • Homepage Click-Through Rate= Did your visitors engage with the site?
  • Homepage Hero Click-Through Rate= Did your visitors engage with your most important promotions?
  • Customer Sign-in Rate= Are we affecting our opportunity to personalize the site experience?

SiteCatalyst can expose the “why” (supporting click actions) as in ‘what are my visitors clicking that explains the change in actions in the new site’. Are they clicking the product imagery on the Homepage which accounts for a higher rate of Product Page Views? Are they clicking the Homepage Hero at a lower rate which explains the reduced conversion rate of the July promotion?

Use these tools in combination to tell the full story of what visitors are doing and how they are getting there. Understanding the user flow by this method explains exactly which elements in your new site is working and which elements may need iterative optimization in a second or third phase of testing.

Segmentation Strategy
And don’t forget to segment! What are your most important audiences? Watch each behavior to ensure you haven’t added any friction in the user flow to one or more groups.

Common segments are:

  • Visitor type: Customers, Prospects
  • Campaign Referrer: Display , email, Paid Search
  • User Device: Desktop, Mobile, Tablet
  • Browser: Firefox, Chrome, Internet Explorer
  • CRM data: DINK, Loyal Customer, At-Risk Customer, Upgrade eligible

Understanding how your most important segments are performing will direct you to your next iterations or subsequent targeting opportunities.

Test Length
Establish a minimum length of time you will run your AB test of the new site. It is often very tempting to call a winner as soon as you see favorable performance but resist the urge! Test for at least one week to gather a full cycle of visitors’ activities. You may see very different behavior or traffic levels on weekends from weekdays.

In addition, watch the cumulative graph of conversion rate to look for plateaus in performance which could indicate that the data has stabilized. If your graph still looks erratic or continues to criss-cross with the other experiences you need to continue to run the test.

If a test coincides with a holiday or promotional period, unusual visitor behavior may require that you extend the test to account for atypical results. You may even push to delay your launch to avoid such risks to your data.

Establish a Clear Launch Plan

Ensure that your project manager has answered at a minimum the following questions:

  • Has the new site been thoroughly QAed?
  • When will the new site be available in production?
  • When will the Test&Target campaign launch? Is it after the new site has been confirmed in production?
  • If the new site needs to be taken down, who can shut off the Test&Target campaign that is directing this traffic?
  • What reporting is required? Who can provide a 24 and 48 hour check-in of performance?
  • Has the the post-launch results review meeting been set?
  • What marketing and technical resources are available within the first 4 hours of launch for questions and to monitor performance?

Determine who owns the test data and who will be interpreting and communicating the performance to leadership and the rest of the team. Make sure you document the details of the test and the performance. In your documentation, clearly define each metric and segment.

Conversion Rate is defined as Approved, Pended, or Declined Applications Submitted.
Photoshop Conversion Rate is defined by the Thank You page firing for ProductID 123 or 234.

Create a clear template to document the performance of the new site. A proactive one-slide summary that includes your hypothesis, site screenshots, performance, and recommendations (continue to run, push new site, redesign sign-in area) will communicate performance at-a-glance and save your inbox and voicemail from being overloaded.

Identify Next Steps

From your data analysis, determine if you can immediately push your new site to 100% of traffic or if first some tweaks are needed. You may find that your site is increasing conversion rate and reducing sign-in rate. Does mean you’ll need to tweak some of the design or will you push to all visitors and continue to test to optimize for sign-ins? Confirm that all analytics and vendor pixels been properly defined and persisted. Are dynamic feeds working as expected or does new code need to be implemented?

Discuss testing the addition of successfully influ­en­tial ele­ments from the pre­vi­ous site to the new site in the immediate weeks that follow. Establish a Roadmap of iterative tests that will follow your new site push based on any secondary metrics that fall short.

Enjoy Your Success

New site launches are really hard work. You’ve probably worked late, checked data early, and confirmed and reconfirmed a thousand steps during the launch process. You’ve worked with tons of folks from different BUs and perhaps several vendors. It’s now time to smile and enjoy the fruits of your hard work.

Testing is the only way to predict the new site’s impact to the business. Congratulations, you’ve been an integral part of providing real value to your company! You’ve either validated the investment in a new digital experience or you’ve helped avoid a huge mistake. Either way you are part of setting the direction of your company’s online channel. Good job and get moving on your next test!

Test­ing is valu­able as well as fun. Happy testing!


About the author: Christi Ter­je­sen is an Opti­miza­tion Man­ager with Adobe Dig­i­tal, a full ser­vice web opti­miza­tion team within Adobe Con­sult­ing. She helps clients in all ver­ti­cals increase con­ver­sion rate and pro­duce ROI through their test­ing pro­grams. She is a cer­tifed Adobe Test&Target expert based out of New York City.