In the movie Jerry Maguire, the title character had to shout “Show me the money!” repeatedly in order to keep his job.  It’s not just a catchy movie quote; it’s also good advice for optimization professionals.

You and I know that testing and optimization are valuable, but your boss (and his/her boss) might not.  In fact, how many people in your organization don’t know about the lift and learning you’re generating through your A/B and multivariate tests?  More importantly, which key people – managers, executives, and decision makers – don’t know?

Well, it’s time to let them know.  It’s time to show them the money!


The Money – “Show me the money!”

There are two parts to showing the money: 1) showing, and 2) money.  In this post we’ll talk about the second part – the money, or monetizing test results.  Next time we’ll discuss the first part – showing, or communicating test results.

Show Me the Money

One of my clients recently said to me, “We just got a 10% lift in revenue per visitor (RPV), but that doesn’t mean anything to many stakeholders.  It’s too nebulous to be meaningful.”  He’s absolutely right.  It’s difficult to fully appreciate the value derived from a test and a testing program when considering percentages and segments.  Furthermore, percentages don’t pay the bills.  Dollars do (or Euros, Pounds, Yen, etc.).  So when we communicate test results to stakeholders, especially key ones, we ought to do it in a very concrete and tangible way – incremental annual revenue.

There are many ways to monetize test results to arrive at an incremental annual revenue figure, some of them more robust and rigorous than others.  One of my clients uses Adobe SiteCatalyst analytics data to help in the calculation.  Another uses scenario analysis to tweak inputs and assumptions and generate a revenue range.  Whichever way you do it, remember that the only thing we know with complete certainty when forecasting is that our forecast will be incorrect.  The point isn’t to predict the future with pinpoint precision; instead, our goal is just to get a ballpark idea.


The Formula –  “You had me at hello.”

With that in mind, let’s take a look at a simple, back-of-the-envelope formula for turning your test results into revenue figures that are easier to understand and communicate.

Incremental RPV  x  Visitors per Day  x  365 days  =  Incremental Annual Revenue

That’s the basic formula; it’s simple and straightforward.  Let’s dive into the two variable inputs:

  • Incremental RPV = Winner RPV – Control RPV
  • Visitors per Day = Total visitors in the campaign / number of days the campaign ran

If you’re like me, Math class made a lot more sense when you had an example, so here we go.  Let’s say you ran a test that produced the results below.  Calculate the incremental annual revenue.  (Show your work for bonus points!)

  • 300,000 visitors
  • 14 days in the campaign
  • Control RPV = $5.00
  • Experience B RPV = $5.35
  • Experience C RPV = $5.50
  • Experience D RPV = $4.90

Incremental RPV  x  Visitors per Day  x  365 days  =  Incremental Annual Revenue

($5.50 – $5.00)  x  (300,000 / 14)  x  365  =  $3,910,714

Congratulations!  You successfully monetized your test results.  Now you know that the winning experience in this test is potentially worth somewhere around $3.9 million in incremental annual revenue.  Now what?  Good question!  More about that next time when we examine how to communicate test results throughout the organization.


The Assumptions –  “Help me help you.”

You know what they say: “When you assume, you make…”  Well, you know the rest.  Unfortunately, when it comes to forecasting and estimating, you have to make some assumptions.  In fact, you have to make a lot of assumptions, and our back-of-the-envelope calculation has some big ones.  Let’s consider a few of them, recognizing that there are even more than those mentioned here.

Visitor Count – We ran the campaign for 14 days and extrapolated annual visitors based on that.  Our assumption is that traffic is perfectly constant.  We know, however, that it is not.  Most sites experience daily fluctuations in traffic, seasonality, and year-over-year growth.

Ceteris Paribus – “All other things being held constant” – When we test, we try to hold everything constant and change only one variable at a time.  That might work for 14 days, but it’s impossible to do for an entire year.  Over time the entire site is going to change, which will undoubtedly affect the winning experience and its incremental RPV.

RPV – Revenue per visitor is easy to calculate for retail organizations; it’s an out-of-the-box metric in A/B testing solutions like Adobe Test&Target.  But what if you have a media, lead gen, or not-for-profit model where RPV isn’t easily measured?  In this case, you have a bit more work to do in order to assign a revenue figure to the key metric you’re tracking.  For lead gen organizations, how much is each lead worth, and how many more leads per visitor did you generate in your winning experience relative to the control?  For media organizations, how much is each ad impression or page view worth, and how many more impressions or page views per visitor did you generate in your winning experience?  With that information in hand, you’ll be able to translate your other metric into RPV.

Whatever your assumptions are, it’s usually a good idea to be somewhat conservative in your estimate (also sometimes affectionately called “sandbagging”).  Even better, put together an estimated range based on several calculations.  You might generate these by running several scenarios, each with a slight change to the assumption set.  Or you might simply take your one figure calculated above and altering that:

  • Original figure = $3.9 million
  • 50% of original = $1.95 million
  • 150% of original = $5.85 million
  • Range = $1.95 million – $5.85 million


Final Thought – “You complete me.”

One last thought – don’t get too wrapped up in monetizing test results that you lose your long-term perspective.  The first time you calculate a really big number from an extremely successful test, it will be very tempting to get caught up in the excitement and start crafting tests to maximize short-term gain.  Remember that the most valuable part of testing is learning about your visitors and your efforts.  What resonates most with visitors?  What types of tests give us the most bang for the buck (see a recent post about efficiency).  Answering these kinds of questions and producing real learning will allow you to generate both short-term and long-term gains.   Monetizing your test results supports that learning process while also allowing you to “show [everyone] the money”, gain support, and test for years to come.