Five Times to Test: 4 — When you spot an opportunity in your analytics

Many folks ini­tially dive into opti­miza­tion because they have an itch to scratch. They have some the­ory they want to test on their land­ing page, or they know their home­page needs an update but are scared that it will make con­ver­sion tank if they roll it out with­out test­ing. Those are great rea­sons to test — and frankly, you can go a loooong time run­ning those same plays over and over, and you will prob­a­bly score repeated goals. If you’ve obvi­ously got the best run­ning back in the league, it’s almost never a bad choice to hand him the ball.

At some point, though, you will likely find that your curios­ity is just buzzing to join up your ana­lyt­ics efforts with your test­ing. And that’s time to test #4: when you spot an oppor­tu­nity in your analytics.

Here’s an exam­ple I per­formed for a large online retailer that was won­der­ing which prod­ucts it should con­sider test­ing out in its home page hero ban­ners. Rather than just run­ning a test of the top rev­enue prod­ucts, wouldn’t it be good to pin­point some key prod­ucts that have sta­tis­ti­cal indi­ca­tors of being more likely to win the test? I thought it would be. So here’s what we decided to do…

We took the 50 or so top-selling prod­ucts and placed them on a scat­ter plot, with Con­ver­sion Rate as the x-axis and Avg. Sell­ing Price as the y-axis. You would nor­mally expect that as price goes up, con­ver­sion goes down because it takes rel­a­tively longer (ie: more vis­its) to research and con­sider a high-priced item before pur­chas­ing. You can fit a trend line to the cen­ter of that scat­ter plot and see that rela­tion­ship quite clearly.

Fig­ure 1:

How­ever, we decided to go out two stan­dard devi­a­tions from that cen­ter trend line and set bound­aries — which by def­i­n­i­tion include 95% of results. This allows us to see the 2.5% of prod­ucts that sta­tis­ti­cally out­per­form the pack on these two met­rics (in green) as well as the 2.5% that sta­tis­ti­cally under­per­form the pack (in red).
The con­clu­sion? We should look at the prod­ucts that out­per­form and ask if we have tested mer­chan­dis­ing them promi­nently through­out the site and in fea­tured adver­tise­ments. If not, we prob­a­bly should be. And those in red? At least we know that we prob­a­bly don’t need to start with those when pick­ing prod­ucts to test in pro­mo­tional ban­ners on the homepage.

Some would say that you need to have a breadth of pric­ing options on the home­page though — you can’t just show­case $1,000 items all over. You need some $50, $100, and $500 items. That’s fine — but don’t make it those two on the bot­tom left. Move hor­i­zon­tally to the right from those and you’ll hit some other prod­uct that has a sim­i­lar price point but a more accept­able con­ver­sion rate.
So, what did we find when we looked at which prod­ucts these four were? Sur­pris­ingly, the green items were not being pro­moted much at all and the red ones were at the top of their cat­e­gory page. On top of that, the green items actu­ally had not just a higher price+conversion combo, they had a much higher gross mar­gin than the red ones. That doesn’t mean the green ones are a slam dunk — but it’s def­i­nitely an indi­ca­tor that you ought to test them out in a few more promi­nent pro­mo­tional loca­tions through­out the site and in your adver­tis­ing — see how they do for you.