Measuring Visitor Engagement: Brave New World or the Emperor’s New Clothes?

Analytics · By On · 7 Comments

Hello, every­one. Here’s a nice meaty post for you to sink your teeth into. And, it will be fol­lowed by sev­eral more, just as meaty, because I’ve decided to focus on one of the most con­tro­ver­sial top­ics in the indus­try right now: audi­ence measurement.

The cred­i­bil­ity of audi­ence mea­sure­ment data from firms like com­Score, Hit­wise and Nielsen has been plagued for years, for a vari­ety of rea­sons, includ­ing fun­da­men­tal lim­i­ta­tions with their data col­lec­tion method­olo­gies, unpre­dictable revi­sions to those method­olo­gies and results from dif­fer­ent firms that con­tra­dict each other.

Recently, some firms have been claim­ing that they have hit on the best and most effec­tive met­ric for online adver­tis­ing: time spent on site.

And I believe they’re dead wrong.

I believe the best way to mea­sure online adver­tis­ing is by the very met­ric we most often use to sell it: by impres­sion. And, while there are a few poten­tial prob­lems in using this as a met­ric, the prob­lems are not insurmountable.

An “impres­sion” is counted any time an ad is “served” to an end-user, and it is the met­ric adver­tis­ers most often use when they pur­chase ad space (the com­monly used “CPM” means cost per thou­sand impres­sions). Yet the met­rics that mea­sure­ment data firms have used in the past have lit­tle to do with impres­sions. Rather, mea­sure­ment firms and the online adver­tis­ing world have exper­i­mented with a num­ber of very dif­fer­ent mea­sure­ment options.

Begin­ning in the late 1990s, we focused on unique vis­i­tors – largely because this was believed to be unique peo­ple. As I’ve dis­cussed in the past, unique vis­i­tors are any­thing but unique peo­ple and this met­ric is fun­da­men­tally flawed. As we grew more sophis­ti­cated, we began to mea­sure how vis­i­tors inter­acted with con­tent on the site. Page views emerged as the de facto stan­dard for mea­sur­ing con­tent con­sump­tion, and for a long time, that met­ric reigned supreme.

But now Web 2.0 tech­nolo­gies are com­pletely dis­rupt­ing the sta­tus quo and the page view is no longer as accu­rate as it once was. With Flash, AJAX and stream­ing media tech­nolo­gies becom­ing more preva­lent, the page view increas­ingly fails to cap­ture the com­plete user expe­ri­ence. That’s because these tech­nolo­gies deliver a richer, more dynamic expe­ri­ence that does not require a new page view with each new piece of content.

The Gap​.com web­site pro­vides a great exam­ple. If you nav­i­gate to the Jeans cat­e­gory, you’ll quickly notice a Flash-based mod­ule in the cen­ter of the page that shows dif­fer­ent jeans on actual peo­ple. It’s pretty slick and you’ll notice as you click on the dif­fer­ent types of jeans, a new per­son will walk out with those jeans on. But you’ll notice some­thing else. Each time you click, the full page does not change. It remains the same, even though the user is able to inter­act dynam­i­cally with fresh con­tent.

Thus, whereas in the past a vis­i­tor would have gone to a sec­ond — or third, or fifth — page in order to get the infor­ma­tion about each sub­se­quent style of jean, she can now remain on a sin­gle page. So in this exam­ple, is the sin­gle “page view” that the vis­i­tor saw really the best mea­sure­ment of that visit, when she actu­ally browsed through sev­eral prod­ucts? Def­i­nitely not.

With this in mind, it makes sense that audi­ence mea­sure­ment firms are explor­ing new ways to quan­tify vis­i­tor engage­ment on the web.

What makes less sense is that they have cho­sen time-spent-on-site as the best mea­sure of audi­ence engage­ment. The move has attracted a good deal of press, some of which has gone so far as to sug­gest this is a ground-breaking step.

In com­ing blog posts, I will go into detail about page views, time spent on site and impres­sions. I’ll explain why time spent on site is not, con­trary to what much of the trade press claims, the best avail­able mea­sure­ment. I will show that, while those met­rics are inter­est­ing prox­ies of ad inven­tory, they are sec­ondary met­rics when it comes to the true nuts and bolts of ad buy­ing, and they do lit­tle or noth­ing to help adver­tis­ers and media pub­lish­ers con­nect with their tar­get audience.

And I’ll prove that mea­sure­ment by impres­sion — the same met­ric that the major­ity of web pub­lish­ers use to sell online ads — is not only desir­able, but is emi­nently attainable.

To me, this deci­sion to mea­sure time spent on site is a case of the Emperor’s new clothes — the Emperor, and nearly every­body else, claims his clothes are the best ever, when in real­ity, he’s wear­ing noth­ing more than air.

Tagged with → • • • •  
  • http://twitter.com/jasonegan Jason Egan

    Engage­ment isn’t really any­thing to begin with. The fact is, they are sell­ing ads, so they should be mea­sur­ing ads. Most peo­ple read­ing this blog post are prob­a­bly aware of the inher­ent flaws in the time spent met­ric. I’m also not a fan of some kind of mashed up com­bi­na­tion of met­rics to cre­ate a cal­cu­lated met­ric called “vis­i­tor engagement.”

    In today’s Web 2.0 world, a vis­i­tor can engage with your brand out­side of your Web site. The fact is, it’s very dif­fi­cult to under­stand total engage­ment. I mean, how do you know if a vis­i­tor is on your site, talk­ing about you on Twit­ter, and par­tic­i­pat­ing in a group on Face­book about your brand? Where does “vis­i­tor engage­ment” start and stop? The panel-based mea­sure­ment ser­vices are try­ing to do exactly this in their ini­tia­tives to track all media con­sump­tion in all places. Unfor­tu­nately, the more that they try this, the more sam­pled their data becomes.

    So to sum up, not a fan of the con­cept of vis­i­tor engage­ment (at least in it’s cur­rent state) here.

  • http://thespec.com danette mcgib­ney

    Well stated, I am look­ing for­ward to more infor­ma­tion from you on this sub­ject. I have read all your arti­cles on Unique Vis­i­tors ver­sus vis­its and have found them very helpfull.

  • http://webponce.com matthew knight

    I await with inter­est your fol­low­ing posts. it is with­out a doubt, a very hot/heated dis­cus­sion point cur­rently, although some of your exam­ples above are flawed. ‘page views’ of course do not work in an ajax or flash envi­ron­ment, but any sen­si­ble devel­oper will have equated page view to con­tent view, or some sim­i­lar mechanic — so whilst the labelling needs to change, view­ing of dis­crete por­tions of con­tent is still rel­e­vant. Time is also rel­e­vant in cer­tain con­texts, for instance brand activ­ity, or con­tent rich sites, but often can be neg­a­tive in more func­tional con­texts. Unique vis­i­tors is of course use­ful to track when you’re try­ing to increase num­bers of users, but again — when you’re track­ing engage­ment, this is irrel­e­vant. I think the ‘time spent on site’ comes from an adver­tis­ing per­spec­tive, where any ‘eye time’ is good, but you’re right, its not the (with a cap­i­tal T) only met­ric. It is all about con­text. As I say, I’m await­ing your com­ing blogs with inter­est, espe­cially as to how you define ‘impres­sion’. Per­son­ally, I hope it means impressed :)

  • http://www.velocitypartners.co.uk Doug Kessler

    Well argued. Mar­ket­ing has always been com­mon sense but in dig­i­tal mar­ket­ing, it’s increas­ingly uncommon.

  • http://blogs.omniture.com/author/mbelkin Matt Belkin

    Matt,

    Thanks for the com­ments. You’re right on in terms of AJAX/Flash — the pageview isn’t as rel­e­vant. But as you also noted, pageviews are fre­quently swapped with a com­pa­ra­ble expe­ri­ence met­ric like con­tent views. And to that point, I believe the chal­lenges of audi­ence mea­sure­ment are only exac­er­bated in Flash/AJAX world, since it’s left up to the audi­ence mea­sure­ment com­pany — not the pub­lisher — as to what con­sti­tutes a “view” and what does not. Stay tuned for the next post and I wel­come your con­tin­ued feedback!

  • http://blogs.omniture.com/author/mbelkin Matt Belkin

    Jason,

    I’m glad you made the point about “vis­i­tor engage­ment” as a mashed up com­bi­na­tion of met­rics. I’ve watched with amuse­ment as a few ven­dors and con­sul­tants have advo­cated a com­plex algo­rithm that allegedly mea­sures “engage­ment”. Per­son­ally I believe it’s a dis­as­trous way to go because it moves you from an objec­tive world to a sub­jec­tive one. It’s essen­tially a huge step back­wards from a data opti­miza­tion stand­point. I’ll blog more about this as soon as I fin­ish this cur­rent series, so stay tuned!

  • http://blog.themarketingpractice.com Lind­say Willott

    I agree with Jason on this one, peo­ple need to stop extrap­o­lat­ing met­rics designed to mea­sure one thing only, out into entire prin­ci­ples like engage­ment. I’m in favour of a three level mea­sure for mar­ket­ing — (1) how much of an activ­ity is being done (2) how effec­tive or effi­cient is that activ­ity and (2) how much value is derived for the pipeline.