In the movie Jerry Maguire, the title char­ac­ter had to shout “Show me the money!” repeat­edly in order to keep his job.  It’s not just a catchy movie quote; it’s also good advice for opti­miza­tion professionals.

You and I know that test­ing and opti­miza­tion are valu­able, but your boss (and his/her boss) might not.  In fact, how many peo­ple in your orga­ni­za­tion don’t know about the lift and learn­ing you’re gen­er­at­ing through your A/B and mul­ti­vari­ate tests?  More impor­tantly, which key peo­ple – man­agers, exec­u­tives, and deci­sion mak­ers – don’t know?

Well, it’s time to let them know.  It’s time to show them the money!


The Money – “Show me the money!”

There are two parts to show­ing the money: 1) show­ing, and 2) money.  In this post we’ll talk about the sec­ond part – the money, or mon­e­tiz­ing test results.  Next time we’ll dis­cuss the first part – show­ing, or com­mu­ni­cat­ing test results.

Show Me the Money

One of my clients recently said to me, “We just got a 10% lift in rev­enue per vis­i­tor (RPV), but that doesn’t mean any­thing to many stake­hold­ers.  It’s too neb­u­lous to be mean­ing­ful.”  He’s absolutely right.  It’s dif­fi­cult to fully appre­ci­ate the value derived from a test and a test­ing pro­gram when con­sid­er­ing per­cent­ages and seg­ments.  Fur­ther­more, per­cent­ages don’t pay the bills.  Dol­lars do (or Euros, Pounds, Yen, etc.).  So when we com­mu­ni­cate test results to stake­hold­ers, espe­cially key ones, we ought to do it in a very con­crete and tan­gi­ble way – incre­men­tal annual revenue.

There are many ways to mon­e­tize test results to arrive at an incre­men­tal annual rev­enue fig­ure, some of them more robust and rig­or­ous than oth­ers.  One of my clients uses Adobe Site­Cat­a­lyst ana­lyt­ics data to help in the cal­cu­la­tion.  Another uses sce­nario analy­sis to tweak inputs and assump­tions and gen­er­ate a rev­enue range.  Whichever way you do it, remem­ber that the only thing we know with com­plete cer­tainty when fore­cast­ing is that our fore­cast will be incor­rect.  The point isn’t to pre­dict the future with pin­point pre­ci­sion; instead, our goal is just to get a ball­park idea.


The For­mula –  “You had me at hello.”

With that in mind, let’s take a look at a sim­ple, back-of-the-envelope for­mula for turn­ing your test results into rev­enue fig­ures that are eas­ier to under­stand and communicate.

Incre­men­tal RPV  x  Vis­i­tors per Day  x  365 days  =  Incre­men­tal Annual Revenue

That’s the basic for­mula; it’s sim­ple and straight­for­ward.  Let’s dive into the two vari­able inputs:

  • Incre­men­tal RPV = Win­ner RPV – Con­trol RPV
  • Vis­i­tors per Day = Total vis­i­tors in the cam­paign / num­ber of days the cam­paign ran

If you’re like me, Math class made a lot more sense when you had an exam­ple, so here we go.  Let’s say you ran a test that pro­duced the results below.  Cal­cu­late the incre­men­tal annual rev­enue.  (Show your work for bonus points!)

  • 300,000 vis­i­tors
  • 14 days in the campaign
  • Con­trol RPV = $5.00
  • Expe­ri­ence B RPV = $5.35
  • Expe­ri­ence C RPV = $5.50
  • Expe­ri­ence D RPV = $4.90

Incre­men­tal RPV  x  Vis­i­tors per Day  x  365 days  =  Incre­men­tal Annual Revenue

($5.50 — $5.00)  x  (300,000 / 14)  x  365  =  $3,910,714

Con­grat­u­la­tions!  You suc­cess­fully mon­e­tized your test results.  Now you know that the win­ning expe­ri­ence in this test is poten­tially worth some­where around $3.9 mil­lion in incre­men­tal annual rev­enue.  Now what?  Good ques­tion!  More about that next time when we exam­ine how to com­mu­ni­cate test results through­out the organization.


The Assump­tions –  “Help me help you.”

You know what they say: “When you assume, you make…”  Well, you know the rest.  Unfor­tu­nately, when it comes to fore­cast­ing and esti­mat­ing, you have to make some assump­tions.  In fact, you have to make a lot of assump­tions, and our back-of-the-envelope cal­cu­la­tion has some big ones.  Let’s con­sider a few of them, rec­og­niz­ing that there are even more than those men­tioned here.

Vis­i­tor Count – We ran the cam­paign for 14 days and extrap­o­lated annual vis­i­tors based on that.  Our assump­tion is that traf­fic is per­fectly con­stant.  We know, how­ever, that it is not.  Most sites expe­ri­ence daily fluc­tu­a­tions in traf­fic, sea­son­al­ity, and year-over-year growth.

Ceteris Paribus – “All other things being held con­stant” – When we test, we try to hold every­thing con­stant and change only one vari­able at a time.  That might work for 14 days, but it’s impos­si­ble to do for an entire year.  Over time the entire site is going to change, which will undoubt­edly affect the win­ning expe­ri­ence and its incre­men­tal RPV.

RPV – Rev­enue per vis­i­tor is easy to cal­cu­late for retail orga­ni­za­tions; it’s an out-of-the-box met­ric in A/B test­ing solu­tions like Adobe Test&Target.  But what if you have a media, lead gen, or not-for-profit model where RPV isn’t eas­ily mea­sured?  In this case, you have a bit more work to do in order to assign a rev­enue fig­ure to the key met­ric you’re track­ing.  For lead gen orga­ni­za­tions, how much is each lead worth, and how many more leads per vis­i­tor did you gen­er­ate in your win­ning expe­ri­ence rel­a­tive to the con­trol?  For media orga­ni­za­tions, how much is each ad impres­sion or page view worth, and how many more impres­sions or page views per vis­i­tor did you gen­er­ate in your win­ning expe­ri­ence?  With that infor­ma­tion in hand, you’ll be able to trans­late your other met­ric into RPV.

What­ever your assump­tions are, it’s usu­ally a good idea to be some­what con­ser­v­a­tive in your esti­mate (also some­times affec­tion­ately called “sand­bag­ging”).  Even bet­ter, put together an esti­mated range based on sev­eral cal­cu­la­tions.  You might gen­er­ate these by run­ning sev­eral sce­nar­ios, each with a slight change to the assump­tion set.  Or you might sim­ply take your one fig­ure cal­cu­lated above and alter­ing that:

  • Orig­i­nal fig­ure = $3.9 million
  • 50% of orig­i­nal = $1.95 million
  • 150% of orig­i­nal = $5.85 million
  • Range = $1.95 mil­lion — $5.85 million


Final Thought – “You com­plete me.”

One last thought – don’t get too wrapped up in mon­e­tiz­ing test results that you lose your long-term per­spec­tive.  The first time you cal­cu­late a really big num­ber from an extremely suc­cess­ful test, it will be very tempt­ing to get caught up in the excite­ment and start craft­ing tests to max­i­mize short-term gain.  Remem­ber that the most valu­able part of test­ing is learn­ing about your vis­i­tors and your efforts.  What res­onates most with vis­i­tors?  What types of tests give us the most bang for the buck (see a recent post about effi­ciency).  Answer­ing these kinds of ques­tions and pro­duc­ing real learn­ing will allow you to gen­er­ate both short-term and long-term gains.   Mon­e­tiz­ing your test results sup­ports that learn­ing process while also allow­ing you to “show [every­one] the money”, gain sup­port, and test for years to come.