New To A/B Testing? A Layout Test Is a Great Place to Start.
As an analytics professional at a media company, you have just received your login to Adobe Test&Target. You nervously enter your pass word and you are in. There before you is a shiny, yet foreign interface. Now what? It took you two years to become a pro with SiteCatalyst and now you feel comfortable quickly cruising through the interface and customized reports you have built. But now you are back at square one with testing, only this time you can really do some damage with your new-found ability to modify content on your site. Where do you start?
Having helped dozens of clients get off the ground and running with their first several tests, I can tell you the question you face is one that challenges most new testers. My advice is to step back and remember that you are not going to boil the ocean all at once. Pick a test that allows you to answer “YES!” to these three important questions:
1. With the resources at your disposal, can you realistically launch this test in a short time?
2. Is the change likely to produce an impact?
3. Will your organization actually implement the change if it proves to be a winner?
In my experience, a great place for media companies to start that meets the three conditions above is a layout test.
Layout tests may not require as many design resources as you think.
These types of layout changes have increased importance for media-based sites, which are generally template driven because the actual content updates frequently. For this reason it is important to focus optimization efforts on the template itself via layout and style changes. Testing the actual content could potentially be very inefficient if the content has a short shelf life.
Layout changes are very visual and are likely to have a large impact.
You can change the font from Arial to Arial Narrow and people may not notice, but if the “Most Popular Videos” module is in the left rail instead of the right rail, it changes how people interact with the page. Maybe visitors tend to scan top-left to bottom-right and your change puts that module in their line of site, or removes it. A test will help you prove that out.
What Happens if I Test a Losing Recipe
You can learn just as much from a losing recipe as a winning recipe. If your challenger recipe produces a 10% decline in site engagement, your test has saved you from making a blind change that would have reduced potential ad revenue by 10% or more (assuming an ad-based CPM model).
A neutral test also tells you something. If changing the layout for the specific template tested has no impact on engagement or success, a layout-intensive redesign may not be the most efficient way to increase engagement. For the section or template tested, you may want to try testing something else less resource intensive that is still likely to produce results, such as the number of “related-articles” to display.
Test Multiple Layouts
Many testers are frustrated when their challenger recipe does not work out because they are at a loss for what to do next, wondering if they should just try a completely new idea. You can significantly increase your chance of finding a winner and learning about your page by testing more than one challenger. In fact, the additional learning you pull out of a test for doing third, fourth, and even fifth recipes will usually far outweigh the effort required to add those additional recipes.
Returning to the “Most Popular Videos” module layout example, if your challenger recipe featuring the module at the bottom of the right rail underperforms your default module at the top of the right rail, you do not know if changing layout was a bad idea, or that particular placement on the right rail was a bad execution of a good idea. Maybe the change pushed the module below the fold. There could be several reasons for the decline, but with only one challenger it can be difficult to drill into what caused the change.
If you test the module at top right (default), top-left (recipe B), bottom-left (recipe C), and bottom-right (recipe D), you are less likely to be thwarted by the bad execution of a good idea and more likely to find something that performs better than what you currently have.
Site optimization through testing can be as complex as you make it. Do not bite off more than you want to chew with your first test. A layout test is a great way to run a simple test that produces an impact.