Never a Failed Test
A while ago I had a bunch of stocks that I was planning on selling. The timing was perfect and the price for the stocks was high. Everything was just as I had planned and hoped for so I made a mental note to do it. As it turns out, someone misplaced my note and I totally forgot to sell the stocks. I remembered a couple days later and went to sell the stocks but the price had dropped a couple dollars. “It will come back up if I give it a day or two” I told myself. A couple days later the price was lower. A few days after that it was even lower. The downward spiral continued for several weeks until the gain I had anticipated had completely vanished.
Over the ensuing months the stock stayed at that low price. There were many times where I felt tempted to just sell anyway—after all, an itty bitty gain is better than no gain, right? Even though I really did want the short term win, I didn’t have any external pressures to keep me from holding on for the long term. Lucky for me I didn’t give in to selling. Fast forward many months and the price had climb back to where it originally had been and I sold the stocks for a nice profit.
What does this have to do with testing and optimization—everything. Good testing is like good investing. The key to consistently winning in the stock market is to be in it for the long run. The same is true for testing. When you are testing for the long run you think and act differently than when you are testing for the short term gain. With a long term approach, you aren’t concerned with the minor ups and downs in lift or the seasonal changes that come and go. You take only marginal notice of competitors and the economy because you know the long term gain is in learning what works for the site as a whole. To put it very simply, when you test for the long run you focus on what you are learning rather than what the lift was.
There are a few ways to gauge how well you and your company are putting learning before lift. I would encourage you to evaluate yourself and your company with each of the following questions.
1. Does every test we run answer a clear business question?
- Remember, when you are testing to learn you design your test to answer a question—not to prove you, your boss, or your department is right.
2. Even before the test launches, do we know the next step for each experience whether it wins or loses?
- Good learning leads to clear iteration. This means you can take what you learned and expound on it to make it even better. When you iterate you follow up on and build upon what you learned. You don’t randomly go from test idea to test idea.
3. Do we see any lift as a good thing, even if it is negative lift?
- Lift doesn’t matter as long as you have it. The worst tests are those that are flat, but if you get negative lift I would be shouting for joy because it means you impacted the visitor experience enough to change their behavior. Changes in visitor behavior usually translate very easily into clear learning.
4. Is testing a process rather than an event?
- If your testing happens in bursts you most likely are not learning. Learning happens a little at a time with consistent steady testing effort. Like the tortoise and the hare, it isn’t the bursts of speed or the marketing blitz that wins the race; it is the consistent steady testing effort.
5. Do we view testing as valuable rather than “risky”?
- When you test to learn you eliminate the idea that testing is “risky” because whether you win or lose, you will learn. Risk is only part of the equation if the test is designed so poorly that you don’t have the right experiences to answer the question or if you are asking the wrong question in the first place.
6. Have we eliminated all external pressures that encourage the “big win” tests?
- I have seen way too many companies shoot for the moon and end up hitting the gutter. Usually the big win tests incorporate so many changes that some results are positive, and some are negative and the overall net is nothing special. When you go for the big win you have no idea what is influencing the success. However, when you consider the massive amounts of time and resources put into these tests, they are always failures.
Hopefully you were able to answer yes to each of the questions above. If not, you may be putting lift before learning and in essence selling your stock prematurely. Focusing on learning is the most valuable thing you can do while testing because you will launch tests that are guaranteed to be winners. When you learn you win. Keep testing and keep your focus on long term learning.