Move Beyond Gut Feel: Testing with Media Engagement Metrics
In my position I have the chance to observe and also assist media companies as they mature towards data driven organizations. It’s an exhilarating time as our industry starts to move past the era of gut feel to a more robust improvement process. For example, some media companies are discovering that the ever popular home page rotating banner is (gasp!) in fact decreasing overall site consumption and impairing the visitor experience. How can that be when almost everyone in the industry uses it? This issue illustrates perfectly that an idea can easily outgrow its own value based in part on the power of “me too” decision making.
That leads me to one question I’ve heard a lot this year, and nothing could make me happier. It goes something like this: ‘Chad, we feel we have a solid analytics and analysis foundation. What’s the next step that will get us X?’ By X I mean any applicable key business requirement critical to the business. In our industry this usually involves increasing onsite consumption or engagement, driving greater loyalty or becoming a stronger top of mind brand for a particular niche. Glad you asked Mr/Ms Media. First, you can ALWAYS improve your analytics and measurement strategy, methods of analysis, adoption etc. I don’t want to minimize that fact by having this discussion.
But, more leading edge media companies are ready and hungry for what I call product agnostic solutions. What does that mean? It simply means using the best combination of products, services and best practices to provide a holistic long term solution to those critical business needs or KBR’s. We need to develop the best solutions that get you to your optimization goals, and increasingly those solutions will be more than just reporting. Analysis is critical to your business, but acting on that analysis and testing the results of those actions in a confidence controlled environment will ultimately move you to a data driven destination. That’s why Omniture has spent so much effort and resources on driving towards an entire optimization suite, not just on SiteCatalyst.
Recently our Test&Target product received a big upgrade that allows it to be a bigger part of product agnostic solutions, an upgrade that was geared specifically for our industry. As a media site, your goal isn’t to efficiently move a visitor through a funnel that leads from landing page to product to shopping cart to purchase confirmation (unless you charge a premium for content-more on that in a later blog post). No, this business requires engagement, and media customers now have engagement specific metrics available in T&T. You can read more about it here. You can now use confidence based testing with truly relevant media metrics such as page views, time on site, and engagement, not just borrowing metrics that apply to other industries.
One media company our consulting team works with has already jumped on these engagement metrics and achieved a “throw an office party” improvement in site consumption. Our consultants introduced this customer to a simple yet powerful testing concept called inclusion/exclusion testing. The idea is to remove individual modules or site functionality from the home page and determine which MISSING modules are most critical to overall site consumption, in this case page views.
A series of testing showed with strong statistical confidence that two page modules had negative impact on consumption when removed, while two others had a positive impact on consumption when removed. In other words the latter two modules caused overall site consumption to go down when they were present on the home page. That represents solid gold information that testing can provide a site experience or content manager beyond the initial analytics analysis. And the best part? This testing concept requires no new creative and minimal technical impact. It’s easy to do!
Change, Change, Change…
But the story doesn’t stop there-and unfortunately this is where many of us would be tempted to declare victory. The next step was to test and implement a more effective combination of modules-to take ACTION using the information. The site utilized an additional series of tests with different combinations of site elements to determine which combinations were most effective together. Again we’re not talking about creating new functionality, just re-considering the existing pieces.
One particular combination, including the modules critical to consumption listed above, drove a 17% documented uplift in overall site consumption. That’s not just click through folks, that’s site wide impact measured in page views! This media company has a CPM driven model and that uplift means a more compelling story to sell to potential advertisers as well as more advertising revenue. Again, this is where analysis derives its true power, in conjunction with statistically measured and documented IMPROVEMENTS when changes are made. Even further, this information will be used to determine which specific user groups change behavior and can be targeted for even greater long term gains.
This is just one of many tests that can be run. Consider some other possibilities for a moment: page reclamation, layout, content affinity testing. Going back to our “gut feel” statement earlier, is the featured content rotating banner (or your other featured content option) really the best way to increase content consumption? Instead of relying on the conventional wisdom or “everyone else is doing it” you can determine with certainty for your business what the best options are, again using critical Media KPI’s (engagement, consumption, time on site). Or, use the integration with your existing SiteCatalyst implementation to test with a custom engagement metric that you’ve already built. There’s no time like the present to build on your analytics foundation by following the data with meaningful action.
What’s on your mind as a media/publishing site? Send me a comment or email. I’d love to hear about it!