A while ago I had a bunch of stocks that I was plan­ning on sell­ing.  The tim­ing was per­fect and the price for the stocks was high.  Every­thing was just as I had planned and hoped for so I made a men­tal note to do it.  As it turns out, some­one mis­placed my note and I totally for­got to sell the stocks.  I remem­bered a cou­ple days later and went to sell the stocks but the price had dropped a cou­ple dol­lars.  “It will come back up if I give it a day or two” I told myself.  A cou­ple days later the price was lower.  A few days after that it was even lower.  The down­ward spi­ral con­tin­ued for sev­eral weeks until the gain I had antic­i­pated had com­pletely vanished.

Over the ensu­ing months the stock stayed at that low price.  There were many times where I felt tempted to just sell anyway—after all, an itty bitty gain is bet­ter than no gain, right?  Even though I really did want the short term win, I didn’t have any exter­nal pres­sures to keep me from hold­ing on for the long term.  Lucky for me I didn’t give in to sell­ing.  Fast for­ward many months and the price had climb back to where it orig­i­nally had been and I sold the stocks for a nice profit.

What does this have to do with test­ing and optimization—everything.  Good test­ing is like good invest­ing.  The key to con­sis­tently win­ning in the stock mar­ket is to be in it for the long run.  The same is true for test­ing.  When you are test­ing for the long run you think and act dif­fer­ently than when you are test­ing for the short term gain.  With a long term approach, you aren’t con­cerned with the minor ups and downs in lift or the sea­sonal changes that come and go.  You take only mar­ginal notice of com­peti­tors and the econ­omy because you know the long term gain is in learn­ing what works for the site as a whole. To put it very sim­ply, when you test for the long run you focus on what you are learn­ing rather than what the lift was.

There are a few ways to gauge how well you and your com­pany are putting learn­ing before lift.  I would encour­age you to eval­u­ate your­self and your com­pany with each of the fol­low­ing questions.

1. Does every test we run answer a clear busi­ness question?

  • Remem­ber, when you are test­ing to learn you design your test to answer a question—not to prove you, your boss, or your depart­ment is right.

2. Even before the test launches, do we know the next step for each expe­ri­ence whether it wins or loses?

  • Good learn­ing leads to clear iter­a­tion. This means you can take what you learned and expound on it to make it even bet­ter.  When you iter­ate you fol­low up on and build upon what you learned.  You don’t ran­domly go from test idea to test idea.

3. Do we see any lift as a good thing, even if it is neg­a­tive lift?

  • Lift doesn’t mat­ter as long as you have it.  The worst tests are those that are flat, but if you get neg­a­tive lift I would be shout­ing for joy because it means you impacted the vis­i­tor expe­ri­ence enough to change their behav­ior.  Changes in vis­i­tor behav­ior usu­ally trans­late very eas­ily into clear learning.

4. Is test­ing a process rather than an event?

  • If your test­ing hap­pens in bursts you most likely are not learn­ing.  Learn­ing hap­pens a lit­tle at a time with con­sis­tent steady test­ing effort.  Like the tor­toise and the hare, it isn’t the bursts of speed or the mar­ket­ing blitz that wins the race; it is the con­sis­tent steady test­ing effort.

5. Do we view test­ing as valu­able rather than “risky”?

  • When you test to learn you elim­i­nate the idea that test­ing is “risky” because whether you win or lose, you will learn.  Risk is only part of the equa­tion if the test is designed so poorly that you don’t have the right expe­ri­ences to answer the ques­tion or if you are ask­ing the wrong ques­tion in the first place.

6. Have we elim­i­nated all exter­nal pres­sures that encour­age the “big win” tests?

  • I have seen way too many com­pa­nies shoot for the moon and end up hit­ting the gut­ter.  Usu­ally the big win tests incor­po­rate so many changes that some results are pos­i­tive, and some are neg­a­tive and the over­all net is noth­ing spe­cial.  When you go for the big win you have no idea what is influ­enc­ing the suc­cess.  How­ever, when you con­sider the mas­sive amounts of time and resources put into these tests, they are always failures.

Hope­fully you were able to answer yes to each of the ques­tions above.  If not, you may be putting lift before learn­ing and in essence sell­ing your stock pre­ma­turely.  Focus­ing on learn­ing is the most valu­able thing you can do while test­ing because you will launch tests that are guar­an­teed to be win­ners.  When you learn you win.  Keep test­ing and keep your focus on long term learning.

Good luck.