A/B/n Testing Strategies When Resources Are Constrained
Continual iterative testing on high-value pages is a powerful optimization strategy for a website. This strategy generally starts with testing large content changes on high-traffic pages and then drills down into testing smaller content sections as value and influence on your primary metric is found. A generalized example of this strategy follows these steps:
- Choose the test’s success metric
- Identify high-traffic and other important pages
- Test the layout by rearranging large content sections
- Test the amount of content on the page by testing the inclusion vs. exclusion of different content blocks.
a. If excluding a content section increases conversion, then leave it off.
b. If excluding a section of content doesn’t hurt or help your primary success metric, then consider changing it to different content or make further testing in this section a low priority.
c. If excluding a section of content decreases conversion, then this has influence on your primary success metric so keep this content on the page and move to the next step below.
- Break content blocks into smaller pieces of content and test each of these smaller sections. If a content block contains a headline, bullet points, image and a call to action link, then test each of those pieces separately.
- Repeat — create new tests and strive to beat the previous test’s winning version
Testing programs are often very resource constrained, making it hard to get the most value out of this strategy. The ideal solution is to staff your testing program to handle the strategy above, but that may not be possible right away. Companies often require programs (optimization programs included) to prove their worth over time before additional headcount/resources are granted. Here are some strategies you can implement as you build up the testing program into an ongoing series of iterative testing across the site.
Mix in ‘easy’ tests
Testing programs can often grind to a halt when IT needs to delay a code release or when the design team is too busy to create the deliverables for your next test. When these delays occur, mix in a few tests that can be launched with zero or minimal effort from those other teams. In your test roadmap, mark tests that can be launched quickly (e.g., tests that simply change headlines, messaging, or images on major pages). Plan ahead and have this content wrapped in an mbox so you can launch the test very quickly. It’s worth noting that ‘easy’ tests don’t necessarily translate to low value. They can be quite powerful too.
Test site changes that are already in the works
If the testing program is so resource constrained that there is not time to create even a test roadmap, at a minimum you can test changes that are already queued up by other teams. Typically new code for these changes has to be created by the developer anyway so it’s not much more effort to make it an A/B test. This is a defensive strategy because it will reveal when those changes hurt conversions and/or revenue. It is better to reveal this loss during a test than to push it out to all site traffic. For positive changes, you will also be able to measure the scope of the improvement, which could inform future site changes.
Good luck and keep striving towards continual iterative testing across your site.