Hello, everyone.Here’s a nice meaty post for you to sink your teeth into. And, it will be followed by several more, just as meaty, because I’ve decided to focus on one of the most controversial topics in the industry right now: audience measurement.
The credibility of audience measurement data from firms like comScore, Hitwise and Nielsen has been plagued for years, for a variety of reasons, including fundamental limitations with their data collection methodologies, unpredictable revisions to those methodologies and results from different firms that contradict each other.
Recently, some firms have been claiming that they have hit on the best and most effective metric for online advertising: time spent on site.
And I believe they’re dead wrong.
I believe the best way to measure online advertising is by the very metric we most often use to sell it: by impression. And, while there are a few potential problems in using this as a metric, the problems are not insurmountable.
An “impression” is counted any time an ad is “served” to an end-user, and it is the metric advertisers most often use when they purchase ad space (the commonly used “CPM” means cost per thousand impressions). Yet the metrics that measurement data firms have used in the past have little to do with impressions. Rather, measurement firms and the online advertising world have experimented with a number of very different measurement options.
Beginning in the late 1990s, we focused on unique visitors – largely because this was believed to be unique people.As I’ve discussed in the past, unique visitors are anything but unique people and this metric is fundamentally flawed.As we grew more sophisticated, we began to measure how visitors interacted with content on the site.Page views emerged as the de facto standard for measuring content consumption, and for a long time, that metric reigned supreme.
But now Web 2.0 technologies are completely disrupting the status quo and the page view is no longer as accurate as it once was. With Flash, AJAX and streaming media technologies becoming more prevalent, the page view increasingly fails to capture the complete user experience. That’s because these technologies deliver a richer, more dynamic experience that does not require a new page view with each new piece of content.
The Gap.com website provides a great example.If you navigate to the Jeans category, you’ll quickly notice a Flash-based module in the center of the page that shows different jeans on actual people.It’s pretty slick and you’ll notice as you click on the different types of jeans, a new person will walk out with those jeans on.But you’ll notice something else.Each time you click, the full page does not change.It remains the same, even though the user is able to interact dynamically with fresh content.
Thus, whereas in the past a visitor would have gone to a second — or third, or fifth — page in order to get the information about each subsequent style of jean, she can now remain on a single page. So in this example, is the single “page view” that the visitor saw really the best measurement of that visit, when she actually browsed through several products? Definitely not.
With this in mind, it makes sense that audience measurement firms are exploring new ways to quantify visitor engagement on the web.
What makes less sense is that they have chosen time-spent-on-site as the best measure of audience engagement.The move has attracted a good deal of press, some of which has gone so far as to suggest this is a ground-breaking step.
In coming blog posts, I will go into detail about page views, time spent on site and impressions. I’ll explain why time spent on site is not, contrary to what much of the trade press claims, the best available measurement. I will show that, while those metrics are interesting proxies of ad inventory, they are secondary metrics when it comes to the true nuts and bolts of ad buying, and they do little or nothing to help advertisers and media publishers connect with their target audience.
And I’ll prove that measurement by impression — the same metric that the majority of web publishers use to sell online ads — is not only desirable, but is eminently attainable.
To me, this decision to measure time spent on site is a case of the Emperor’s new clothes — the Emperor, and nearly everybody else, claims his clothes are the best ever, when in reality, he’s wearing nothing more than air.
Engagement isn’t really anything to begin with. The fact is, they are selling ads, so they should be measuring ads. Most people reading this blog post are probably aware of the inherent flaws in the time spent metric. I’m also not a fan of some kind of mashed up combination of metrics to create a calculated metric called “visitor engagement.”
In today’s Web 2.0 world, a visitor can engage with your brand outside of your Web site. The fact is, it’s very difficult to understand total engagement. I mean, how do you know if a visitor is on your site, talking about you on Twitter, and participating in a group on Facebook about your brand? Where does “visitor engagement” start and stop? The panel-based measurement services are trying to do exactly this in their initiatives to track all media consumption in all places. Unfortunately, the more that they try this, the more sampled their data becomes.
So to sum up, not a fan of the concept of visitor engagement (at least in it’s current state) here.
http://thespec.com danette mcgibney
Well stated, I am looking forward to more information from you on this subject. I have read all your articles on Unique Visitors versus visits and have found them very helpfull.
http://webponce.com matthew knight
I await with interest your following posts. it is without a doubt, a very hot/heated discussion point currently, although some of your examples above are flawed. ‘page views’ of course do not work in an ajax or flash environment, but any sensible developer will have equated page view to content view, or some similar mechanic — so whilst the labelling needs to change, viewing of discrete portions of content is still relevant. Time is also relevant in certain contexts, for instance brand activity, or content rich sites, but often can be negative in more functional contexts. Unique visitors is of course useful to track when you’re trying to increase numbers of users, but again — when you’re tracking engagement, this is irrelevant. I think the ‘time spent on site’ comes from an advertising perspective, where any ‘eye time’ is good, but you’re right, its not the (with a capital T) only metric. It is all about context. As I say, I’m awaiting your coming blogs with interest, especially as to how you define ‘impression’. Personally, I hope it means impressed
http://www.velocitypartners.co.uk Doug Kessler
Well argued. Marketing has always been common sense but in digital marketing, it’s increasingly uncommon.
http://blogs.omniture.com/author/mbelkin Matt Belkin
Matt,
Thanks for the comments. You’re right on in terms of AJAX/Flash — the pageview isn’t as relevant. But as you also noted, pageviews are frequently swapped with a comparable experience metric like content views. And to that point, I believe the challenges of audience measurement are only exacerbated in Flash/AJAX world, since it’s left up to the audience measurement company — not the publisher — as to what constitutes a “view” and what does not. Stay tuned for the next post and I welcome your continued feedback!
http://blogs.omniture.com/author/mbelkin Matt Belkin
Jason,
I’m glad you made the point about “visitor engagement” as a mashed up combination of metrics. I’ve watched with amusement as a few vendors and consultants have advocated a complex algorithm that allegedly measures “engagement”. Personally I believe it’s a disastrous way to go because it moves you from an objective world to a subjective one. It’s essentially a huge step backwards from a data optimization standpoint. I’ll blog more about this as soon as I finish this current series, so stay tuned!
I agree with Jason on this one, people need to stop extrapolating metrics designed to measure one thing only, out into entire principles like engagement. I’m in favour of a three level measure for marketing — (1) how much of an activity is being done (2) how effective or efficient is that activity and (2) how much value is derived for the pipeline.
Measuring Visitor Engagement: Brave New World or the Emperor’s New Clothes?
Hello, everyone. Here’s a nice meaty post for you to sink your teeth into. And, it will be followed by several more, just as meaty, because I’ve decided to focus on one of the most controversial topics in the industry right now: audience measurement.
The credibility of audience measurement data from firms like comScore, Hitwise and Nielsen has been plagued for years, for a variety of reasons, including fundamental limitations with their data collection methodologies, unpredictable revisions to those methodologies and results from different firms that contradict each other.
Recently, some firms have been claiming that they have hit on the best and most effective metric for online advertising: time spent on site.
And I believe they’re dead wrong.
I believe the best way to measure online advertising is by the very metric we most often use to sell it: by impression. And, while there are a few potential problems in using this as a metric, the problems are not insurmountable.
An “impression” is counted any time an ad is “served” to an end-user, and it is the metric advertisers most often use when they purchase ad space (the commonly used “CPM” means cost per thousand impressions). Yet the metrics that measurement data firms have used in the past have little to do with impressions. Rather, measurement firms and the online advertising world have experimented with a number of very different measurement options.
Beginning in the late 1990s, we focused on unique visitors – largely because this was believed to be unique people. As I’ve discussed in the past, unique visitors are anything but unique people and this metric is fundamentally flawed. As we grew more sophisticated, we began to measure how visitors interacted with content on the site. Page views emerged as the de facto standard for measuring content consumption, and for a long time, that metric reigned supreme.
But now Web 2.0 technologies are completely disrupting the status quo and the page view is no longer as accurate as it once was. With Flash, AJAX and streaming media technologies becoming more prevalent, the page view increasingly fails to capture the complete user experience. That’s because these technologies deliver a richer, more dynamic experience that does not require a new page view with each new piece of content.
The Gap.com website provides a great example. If you navigate to the Jeans category, you’ll quickly notice a Flash-based module in the center of the page that shows different jeans on actual people. It’s pretty slick and you’ll notice as you click on the different types of jeans, a new person will walk out with those jeans on. But you’ll notice something else. Each time you click, the full page does not change. It remains the same, even though the user is able to interact dynamically with fresh content.
Thus, whereas in the past a visitor would have gone to a second — or third, or fifth — page in order to get the information about each subsequent style of jean, she can now remain on a single page. So in this example, is the single “page view” that the visitor saw really the best measurement of that visit, when she actually browsed through several products? Definitely not.
With this in mind, it makes sense that audience measurement firms are exploring new ways to quantify visitor engagement on the web.
What makes less sense is that they have chosen time-spent-on-site as the best measure of audience engagement. The move has attracted a good deal of press, some of which has gone so far as to suggest this is a ground-breaking step.
In coming blog posts, I will go into detail about page views, time spent on site and impressions. I’ll explain why time spent on site is not, contrary to what much of the trade press claims, the best available measurement. I will show that, while those metrics are interesting proxies of ad inventory, they are secondary metrics when it comes to the true nuts and bolts of ad buying, and they do little or nothing to help advertisers and media publishers connect with their target audience.
And I’ll prove that measurement by impression — the same metric that the majority of web publishers use to sell online ads — is not only desirable, but is eminently attainable.
To me, this decision to measure time spent on site is a case of the Emperor’s new clothes — the Emperor, and nearly everybody else, claims his clothes are the best ever, when in reality, he’s wearing nothing more than air.