Five Times to Test: 4 — When you spot an opportunity in your analytics
Many folks initially dive into optimization because they have an itch to scratch. They have some theory they want to test on their landing page, or they know their homepage needs an update but are scared that it will make conversion tank if they roll it out without testing. Those are great reasons to test — and frankly, you can go a loooong time running those same plays over and over, and you will probably score repeated goals. If you’ve obviously got the best running back in the league, it’s almost never a bad choice to hand him the ball.
At some point, though, you will likely find that your curiosity is just buzzing to join up your analytics efforts with your testing. And that’s time to test #4: when you spot an opportunity in your analytics.
Here’s an example I performed for a large online retailer that was wondering which products it should consider testing out in its home page hero banners. Rather than just running a test of the top revenue products, wouldn’t it be good to pinpoint some key products that have statistical indicators of being more likely to win the test? I thought it would be. So here’s what we decided to do…
We took the 50 or so top-selling products and placed them on a scatter plot, with Conversion Rate as the x-axis and Avg. Selling Price as the y-axis. You would normally expect that as price goes up, conversion goes down because it takes relatively longer (ie: more visits) to research and consider a high-priced item before purchasing. You can fit a trend line to the center of that scatter plot and see that relationship quite clearly.
However, we decided to go out two standard deviations from that center trend line and set boundaries — which by definition include 95% of results. This allows us to see the 2.5% of products that statistically outperform the pack on these two metrics (in green) as well as the 2.5% that statistically underperform the pack (in red).
The conclusion? We should look at the products that outperform and ask if we have tested merchandising them prominently throughout the site and in featured advertisements. If not, we probably should be. And those in red? At least we know that we probably don’t need to start with those when picking products to test in promotional banners on the homepage.
Some would say that you need to have a breadth of pricing options on the homepage though — you can’t just showcase $1,000 items all over. You need some $50, $100, and $500 items. That’s fine — but don’t make it those two on the bottom left. Move horizontally to the right from those and you’ll hit some other product that has a similar price point but a more acceptable conversion rate.
So, what did we find when we looked at which products these four were? Surprisingly, the green items were not being promoted much at all and the red ones were at the top of their category page. On top of that, the green items actually had not just a higher price+conversion combo, they had a much higher gross margin than the red ones. That doesn’t mean the green ones are a slam dunk — but it’s definitely an indicator that you ought to test them out in a few more prominent promotional locations throughout the site and in your advertising — see how they do for you.