Many folks ini­tially dive into opti­miza­tion because they have an itch to scratch. They have some the­ory they want to test on their land­ing page, or they know their home­page needs an update but are scared that it will make con­ver­sion tank if they roll it out with­out test­ing. Those are great rea­sons to test — and frankly, you can go a loooong time run­ning those same plays over and over, and you will prob­a­bly score repeated goals. If you’ve obvi­ously got the best run­ning back in the league, it’s almost never a bad choice to hand him the ball.

At some point, though, you will likely find that your curios­ity is just buzzing to join up your ana­lyt­ics efforts with your test­ing. And that’s time to test #4: when you spot an oppor­tu­nity in your analytics.

Here’s an exam­ple I per­formed for a large online retailer that was won­der­ing which prod­ucts it should con­sider test­ing out in its home page hero ban­ners. Rather than just run­ning a test of the top rev­enue prod­ucts, wouldn’t it be good to pin­point some key prod­ucts that have sta­tis­ti­cal indi­ca­tors of being more likely to win the test? I thought it would be. So here’s what we decided to do…

We took the 50 or so top-selling prod­ucts and placed them on a scat­ter plot, with Con­ver­sion Rate as the x-axis and Avg. Sell­ing Price as the y-axis. You would nor­mally expect that as price goes up, con­ver­sion goes down because it takes rel­a­tively longer (ie: more vis­its) to research and con­sider a high-priced item before pur­chas­ing. You can fit a trend line to the cen­ter of that scat­ter plot and see that rela­tion­ship quite clearly.

Fig­ure 1:

How­ever, we decided to go out two stan­dard devi­a­tions from that cen­ter trend line and set bound­aries — which by def­i­n­i­tion include 95% of results. This allows us to see the 2.5% of prod­ucts that sta­tis­ti­cally out­per­form the pack on these two met­rics (in green) as well as the 2.5% that sta­tis­ti­cally under­per­form the pack (in red).
The con­clu­sion? We should look at the prod­ucts that out­per­form and ask if we have tested mer­chan­dis­ing them promi­nently through­out the site and in fea­tured adver­tise­ments. If not, we prob­a­bly should be. And those in red? At least we know that we prob­a­bly don’t need to start with those when pick­ing prod­ucts to test in pro­mo­tional ban­ners on the homepage.

Some would say that you need to have a breadth of pric­ing options on the home­page though — you can’t just show­case $1,000 items all over. You need some $50, $100, and $500 items. That’s fine — but don’t make it those two on the bot­tom left. Move hor­i­zon­tally to the right from those and you’ll hit some other prod­uct that has a sim­i­lar price point but a more accept­able con­ver­sion rate.
So, what did we find when we looked at which prod­ucts these four were? Sur­pris­ingly, the green items were not being pro­moted much at all and the red ones were at the top of their cat­e­gory page. On top of that, the green items actu­ally had not just a higher price+conversion combo, they had a much higher gross mar­gin than the red ones. That doesn’t mean the green ones are a slam dunk — but it’s def­i­nitely an indi­ca­tor that you ought to test them out in a few more promi­nent pro­mo­tional loca­tions through­out the site and in your adver­tis­ing — see how they do for you.

Brig Graff
Brig Graff

That's right Indu. It's been a while since I built it, so don't think I could give you a play-by-play on how to do it, but I'm sure that using the trendlines feature in the current version of Excel you can. Best of luck!

Indu Sriram
Indu Sriram

Hi Brig, Great post! I was curious as to how you created the chart? What was the software that you used? I thought it was excel with the Data Analysis add in. Thanks