Move Beyond Gut Feel: Testing with Media Engagement Metrics

In my posi­tion I have the chance to observe and also assist media com­pa­nies as they mature towards data dri­ven orga­ni­za­tions. It’s an exhil­a­rat­ing time as our indus­try starts to move past the era of gut feel to a more robust improve­ment process. For exam­ple, some media com­pa­nies are dis­cov­er­ing that the ever pop­u­lar home page rotat­ing ban­ner is (gasp!) in fact decreas­ing over­all site con­sump­tion and impair­ing the vis­i­tor expe­ri­ence. How can that be when almost every­one in the indus­try uses it? This issue illus­trates per­fectly that an idea can eas­ily out­grow its own value based in part on the power of “me too” deci­sion making.

That leads me to one ques­tion I’ve heard a lot this year, and noth­ing could make me hap­pier. It goes some­thing like this: ‘Chad, we feel we have a solid ana­lyt­ics and analy­sis foun­da­tion. What’s the next step that will get us X?’ By X I mean any applic­a­ble key busi­ness require­ment crit­i­cal to the busi­ness. In our indus­try this usu­ally involves increas­ing onsite con­sump­tion or engage­ment, dri­ving greater loy­alty or becom­ing a stronger top of mind brand for a par­tic­u­lar niche. Glad you asked Mr/Ms Media. First, you can ALWAYS improve your ana­lyt­ics and mea­sure­ment strat­egy, meth­ods of analy­sis, adop­tion etc. I don’t want to min­i­mize that fact by hav­ing this discussion.

But, more lead­ing edge media com­pa­nies are ready and hun­gry for what I call prod­uct agnos­tic solu­tions. What does that mean? It sim­ply means using the best com­bi­na­tion of prod­ucts, ser­vices and best prac­tices to pro­vide a holis­tic long term solu­tion to those crit­i­cal busi­ness needs or KBR’s. We need to develop the best solu­tions that get you to your opti­miza­tion goals, and increas­ingly those solu­tions will be more than just report­ing. Analy­sis is crit­i­cal to your busi­ness, but act­ing on that analy­sis and test­ing the results of those actions in a con­fi­dence con­trolled envi­ron­ment will ulti­mately move you to a data dri­ven des­ti­na­tion. That’s why Omni­ture has spent so much effort and resources on dri­ving towards an entire opti­miza­tion suite, not just on SiteCatalyst.

Recently our Test&Target prod­uct received a big upgrade that allows it to be a big­ger part of prod­uct agnos­tic solu­tions, an upgrade that was geared specif­i­cally for our indus­try. As a media site, your goal isn’t to effi­ciently move a vis­i­tor through a fun­nel that leads from land­ing page to prod­uct to shop­ping cart to pur­chase con­fir­ma­tion (unless you charge a pre­mium for content-more on that in a later blog post). No, this busi­ness requires engage­ment, and media cus­tomers now have engage­ment spe­cific met­rics avail­able in T&T. You can read more about it here. You can now use con­fi­dence based test­ing with truly rel­e­vant media met­rics such as page views, time on site, and engage­ment, not just bor­row­ing met­rics that apply to other industries.

One media com­pany our con­sult­ing team works with has already jumped on these engage­ment met­rics and achieved a “throw an office party” improve­ment in site con­sump­tion. Our con­sul­tants intro­duced this cus­tomer to a sim­ple yet pow­er­ful test­ing con­cept called inclusion/exclusion test­ing. The idea is to remove indi­vid­ual mod­ules or site func­tion­al­ity from the home page and deter­mine which MISSING mod­ules are most crit­i­cal to over­all site con­sump­tion, in this case page views.

A series of test­ing showed with strong sta­tis­ti­cal con­fi­dence that two page mod­ules had neg­a­tive impact on con­sump­tion when removed, while two oth­ers had a pos­i­tive impact on con­sump­tion when removed. In other words the lat­ter two mod­ules caused over­all site con­sump­tion to go down when they were present on the home page. That rep­re­sents solid gold infor­ma­tion that test­ing can pro­vide a site expe­ri­ence or con­tent man­ager beyond the ini­tial ana­lyt­ics analy­sis. And the best part? This test­ing con­cept requires no new cre­ative and min­i­mal tech­ni­cal impact. It’s easy to do!

Change, Change, Change…

But the story doesn’t stop there-and unfor­tu­nately this is where many of us would be tempted to declare vic­tory. The next step was to test and imple­ment a more effec­tive com­bi­na­tion of modules-to take ACTION using the infor­ma­tion. The site uti­lized an addi­tional series of tests with dif­fer­ent com­bi­na­tions of site ele­ments to deter­mine which com­bi­na­tions were most effec­tive together. Again we’re not talk­ing about cre­at­ing new func­tion­al­ity, just re-considering the exist­ing pieces.

One par­tic­u­lar com­bi­na­tion, includ­ing the mod­ules crit­i­cal to con­sump­tion listed above, drove a 17% doc­u­mented uplift in over­all site con­sump­tion. That’s not just click through folks, that’s site wide impact mea­sured in page views!  This media com­pany has a CPM dri­ven model and that uplift means a more com­pelling story to sell to poten­tial adver­tis­ers as well as more adver­tis­ing rev­enue. Again, this is where analy­sis derives its true power, in con­junc­tion with sta­tis­ti­cally mea­sured and doc­u­mented IMPROVEMENTS when changes are made. Even fur­ther, this infor­ma­tion will be used to deter­mine which spe­cific user groups change behav­ior and can be tar­geted for even greater long term gains.

This is just one of many tests that can be run. Con­sider some other pos­si­bil­i­ties for a moment: page recla­ma­tion, lay­out, con­tent affin­ity test­ing. Going back to our “gut feel” state­ment ear­lier, is the fea­tured con­tent rotat­ing ban­ner (or your other fea­tured con­tent option) really the best way to increase con­tent con­sump­tion? Instead of rely­ing on the con­ven­tional wis­dom or “every­one else is doing it” you can deter­mine with cer­tainty for your busi­ness what the best options are, again using crit­i­cal Media KPI’s (engage­ment, con­sump­tion, time on site). Or, use the inte­gra­tion with your exist­ing Site­Cat­a­lyst imple­men­ta­tion to test with a cus­tom engage­ment met­ric that you’ve already built. There’s no time like the present to build on your ana­lyt­ics foun­da­tion by fol­low­ing the data with mean­ing­ful action.

What’s on your mind as a media/publishing site? Send me a com­ment or email. I’d love to hear about it!