What should I test?

There are lots of ways to approaching testing and optimization. You can be more opportunistic and test pages or elements ad hoc. There can be some success, but it’s often by chance. Being more methodical about what you test and where you test can yield much higher returns. Assuming you’ve got an overall strategy in place, you need to determine what you should test and where. (If you haven’t got the foundations yet, take a look at my previous post on Taking optimization from strategy to execution.)

Where to test

Take a look at your sites top pages for traffic and conversion. If a lot of visitors are going to these pages, there must be something worth looking at. Your home page is a good start, but depending on what line of business you’re in, it might be tough to know what visitors’ intent is. At Adobe, we get millions of visitors a month coming to our site just to download the Flash plugin or Adobe Reader. Others come because they recalled advertising they saw and just typed in the URL.

You can leverage search keywords to see what landing pages or preferred landing pages the visitors have been driven to, and leverage those pages as well.

Test types  

There are lots of types of things to test:

  • Content
  • Appearance (creative execution)
  • Layout
  • Functionality
  • Existence (does having an element on a page matter)

Content and appearance are the easiest things to test. As long as you have the means to have well written content and/or well executed design, it’s fairly easy to implement.

Layout and functionality require a bit more work. And by someone who has web development skills.

Existence falls somewhere in the middle in terms of level of effort. It really depends on the page and it depends on what you’re removing or adding in.

A methodical approach 

It’s great to know all these different types of tests you can do, but now what? Ideally, you’d want to find out first which elements make a difference or do an existence test. Surprisingly on some of our web pages, we have actually experimented with removing the purchase “pod” to see if it makes a difference. In some cases, it actually doesn’t, so we’ve taken it off the page to clear up the clutter.

If you’ve determined you have the right elements, you can try moving the elements around (layout test) or treat them differently creatively (appearance test).

Lastly, and certainly easiest, you can experiment with content. You can try different positioning, specific words to define a feature of your product, or maybe experiment with catchy headlines.

Functionality is by far the toughest thing to test. And because it is, you have to have a pretty good reason to want to. Do you have access to usability studies or customer satisfaction reports from your site? These are good indicators if the user experience is amiss and you need to intervene.

You know your business best

Ultimately, you might find yourself weaving in and out of these different types of tests because of opportunities that present themselves. As long as you can continue to learn while you approach your testing program, you are well on your way.

Taking optimization from strategy to execution

I’m often asked how did we get ramped up to doing 3-5 tests a week. I have to admit, I’m impressed that we’ve gotten this far, but we still have room to grow.

In short, it comes down to 3 things to get you there: aligning your optimization strategies with your business goals, establishing the right framework and processes, and executing. I’ll break it down.

Align your optimization strategies with your business goals

It may seem obvious, but there’s often the impulse to just start testing everything everywhere. You’re enthusiastic and want to show ROI immediately, but you need a strategy. What are your company’s goals? Most often there are revenue goals you’re looking to meet and there are business strategies you need to tap into to help get you there. Is there messaging you can experiment with to support the business strategies? Creative expression? Layout? Functionality? Answering these questions can help guide you.

Establish the right framework and processes 

It’s no question that if your company isn’t doing testing yet, it’s going to take a little time to shift the mindset; it’s a culture change. Depending on how open teams are to change and taking your business to the next level, will determine how quickly you can implement a program. You’ll need:

  • Operating model alignment: You’ll have to get buy in across your organization. It might mean asking other teams to support your efforts or ask permission to take on some other tasks like designing pages.
  • Process design: It’s going to depend on your organization how you want to work. I’ll elaborate more in the following paragraph.
  • Change impact and readiness: How does adding testing affect other teams?
  • Organizational design: Do you keep it all in one group or do you share responsibilities with other teams?
  • End user training: Does the team have the skills they need?

If you’re doing it all yourself, you don’t have to work about processes as much. However, if you’re working across divisions or departments it’s best to have roles, responsibilities, and processes established and agreed upon. You’ll need to secure:

  • Governance
  • Web strategist
  • Optimization manager
  • Campaign enablement
  • Analytics support
  • Web production
  • Project management
  • Web development (for more sophisticated testing)
  • Demand and media generation (to drive traffic to your site)
  • Creative development
  • User experience expertise
  • Platform architecture

Execute your tests 

Now you have your strategies, a slew of test ideas which support your strategies and business goals, you have your team in place, now you have to execute. How do you know what to test first? The first rule of thumb is analyze which pages on your site yield the highest conversion, have the highest volume of traffic, or demonstrate the biggest area of opportunity. We’ve set up scoring models to help us with prioritization. As the team grows, you have to manage lots of tests getting ready to launch. Potentially all at the same time.

Our scoring model is simple. What type of test is it: Content, layout, functionality, or creative? What’s the level of effort: Low, medium, or high? What is the potential revenue lift: Low, medium, or high? The test type score is based on previous quarter’s successes. Have an analyst help you with that. The level of effort is judged by the person doing the enablement of the test which might include development. And the potential revenue impact is also evaluated by an analyst.

The optimization manager should develop their strategy and then come up with individual tests to support those strategies. Then methodically develop test charters or plans which outline the goals of the test, justification, what page(s) the test should be run on. It’s good to have a hypothesis, but be cautious on including that in your test charter because it might sway you to want certain results.

Set up your test using a solution like Adobe Target, run it 3-5 weeks (depending on traffic and conversion rates), and evaluate the success through the reports in the tool or if you want to dig deeper, use something like Adobe Analtyics. Use what you learn to feed your next test idea and keep iterating.

Watch your conversions increase 

The more comfortable you get with testing, the quicker you can iterate on tests and reap the rewards of increased conversions. Happy testing!


Lisa Flosznik manages testing and optimization for the Digital Media business at Adobe.