Testing Resolutions for 2013
With a new year comes the opportunity to review what we accomplished the previous year and ponder what we hope to achieve in the new year. In an effort to reconcile that difference we often make resolutions for what we will change.
“I will read one book each month.“
“I will go to the gym twice a week.“
“I will call my mother once a week.”
While we start strong in January, by the end of March we’ve read the prologue to Moby Dick, the sight of the gym on the way to work reminds us of the expensive membership we’re not using, and mom won’t stop calling to ask why we’re not calling.
In an effort to help you make resolutions you can keep, I have compiled a short list of testing and optimization resolutions. After working with dozens of testing organizations, I have had the opportunity to think about where I’ve seen testers succeed and fail. Below are six easy resolutions you can keep that will make a difference to your testing program.
1. “I won’t focus on tracking clicks. Instead I will focus on revenue.”
If revenue comes from a subscription confirmation, I will focus on confirmations. If I have a CPM ad-based model, I will focus on driving page views and ad impressions. I will not let my focus shift to secondary metrics that have only correlative relationships with revenue.
2. “I will focus on tests that generate learning instead of just tests that produce winners.”
Finding a winning test recipe is great in that it elevates you from point A to point B, but some learning tests will help you understand how the change impacts the page. This learning may help create a roadmap that can move you beyond point B to a point E or F you haven’t considered.
3. “I will not waste my time trying to get my testing numbers to match my analytics numbers.”
I’ve said quite a bit about this in the past so I won’t dwell on this for long. While it’s comforting to see your visitor counts in a testing campaign line up well with your analytics data during the same period, that’s not important. What is important is that your testing data allows you to see comparative rates of change between the test cells. A 10% lift is the same rate of change regardless of if the base conversion rate is 1.4% or 14%.
4. “I will add segments to all my tests.”
A test with no segments gives you a limited, one-dimensional view of how the test cells do in aggregate. When you add segments, you can slice and dice the data to see what resonates with key groups of visitors. Does traffic from my top five referrers respond differently than traffic from everywhere else? Are there specific page layouts that perform better for paid search visitors than for organic search visitors? Can browser segments help me catch a slight technical glitch in Internet Explorer early on before I let that glitch ruin my entire test? In Test&Target segments are quick and easy to set up, but without them, all of your data goes into one pile and there is no way to split it back out again.
5. “I will not run multiple campaigns on the same page at the same time.”
When a visitor falls into two campaigns on the same page, or in the same flow and they share the same success metrics you create an attribution problem. Was it the changes in campaign A or campaign B that really had the impact? This kind of crossover creates undesirable noise in the data and can be avoided if you make the campaigns mutually exclusive by running them at different times or splitting your audience into exclusive testing groups such that a visitor can only land in one campaign or the other.
6. “I won’t ask my Test&Target consultant how to measure bounce rate.”
This is a topic and discussion I have had with many testers. I suppose it is a result of so many people shifting into a testing role having come from an analytics role. While bounce rate is a key metric in the correlation-driven analytics world, its value is questionable in the testing world where the cause-and-effect relationship between a change the desired behavior is key. A tested change to a key landing page could do a better job of filtering out unqualified traffic, while increasing the volume of qualified traffic through to your ultimate goal. In this scenario your conversion rate may improve even as bounce rate increases. Resolve to not use bounce rate as a proxy for something else you can measure that is directly tied to revenue.
Testing is a learning process. Take some time and ponder how you can incorporate these resolutions in your optimization program. Just as your personal New Year’s resolutions provide an often-needed kick start to the year, you may find these testing resolutions breathe new life into your testing program — and they don’t require 2-year gym membership commitment. Keep at it and don’t give up in February.