Do you use mul­ti­ple mea­sure­ment sys­tems to get a han­dle on your web­site traf­fic? For exam­ple, a online busi­ness opti­miza­tion or web ana­lyt­ics plat­form like Omni­ture Site­Cat­a­lyst along­side a panel-based audi­ence mea­sure­ment ser­vice like Com­score Media Metrix, Hit­wise, or Nielsen NetRat­ings? If so, you’ve undoubt­edly won­dered why the met­rics reported by these sys­tems often don’t line up. In par­tic­u­lar, why are unique vis­i­tors often dif­fer­ent? How can one sys­tem report 5 mil­lion unique vis­i­tors per month, and the other sys­tem report 2 million?!

These are among the most fre­quently asked ques­tions I hear from media and con­tent cus­tomers. But this issue extends well beyond media com­pa­nies. As it turns out, many retail­ers, busi­ness to busi­ness, and travel web­sites also use web ana­lyt­ics and audi­ence mea­sure­ment ser­vices side by side. And wher­ever these two plat­forms co-exist, there are always ques­tions as to why the num­bers do not line up.

So in this new blog series, I wanted to share my own thoughts on the mat­ter. And per­haps most impor­tantly, I wanted to offer some guide­lines on when and why you should use one set of fig­ures vs. another.

COMPARING APPLES TO ORANGES
To begin with, it’s crit­i­cal to under­stand that online busi­ness opti­miza­tion or web ana­lyt­ics (i.e. Omni­ture Site­Cat­a­lyst) and panel-based audi­ence mea­sure­ment (i.e. Com­score Media Metrix, Hit­wise, Nielsen) use fun­da­men­tally dif­fer­ent mea­sure­ment method­olo­gies. Even though they use sim­i­lar indus­try terms, such as unique vis­i­tors, you’re really com­par­ing apples to oranges if you try to com­pare both sets of num­bers. Some of you already know this, but many peo­ple actu­ally do not real­ize how fun­da­men­tally dif­fer­ent they are.

WEB ANALYTICS AND UNIQUE VISITORS: DEMYSTIFIED
To demys­tify these method­olo­gies, let’s start with web ana­lyt­ics. In gen­eral, web ana­lyt­ics or busi­ness opti­miza­tion plat­forms like Omni­ture Site­Cat­a­lyst mea­sure online activ­ity at the vis­i­tor level for roughly 100% of your unique vis­i­tors. How­ever, as I’ve high­lighted in pre­vi­ous posts, unique vis­i­tors is a fairly loose term in web ana­lyt­ics. As it relates to this dis­cus­sion, one of the most fun­da­men­tal dif­fer­ences is how web ana­lyt­ics pack­ages typ­i­cally mea­sure unique visitors.

Specif­i­cally, web ana­lyt­ics plat­forms typ­i­cally define a unique vis­i­tor as: a unique cookie set on a unique com­puter and a unique browser that accesses your website.

So there are three key pieces there — unique cookie, unique com­puter, and unique browser. If you change any one of these, you get a new “unique vis­i­tor”. For exam­ple, if you use Inter­net Explorer to visit one web­site, then switch to Fire­fox — this counts as two unique vis­i­tors. Sim­i­larly, visit a web­site from work and then from home — two unique vis­i­tors. Your kids visit a web­site from your com­puter, you visit the same web­site from your com­puter — that’s one unique vis­i­tor, even though it should arguably be two. This is really impor­tant to under­stand, and one of the main rea­sons you need to be cau­tious in using unique vis­i­tors as a strate­gic metric.

AUDIENCE MEASUREMENT AND UNIQUE VISITORS: DEMYSTIFIED
Now let’s look at audi­ence mea­sure­ment and how those ser­vices typ­i­cally mea­sure unique vis­i­tors. Audi­ence mea­sure­ment ser­vices gen­er­ally lever­age a sam­pling method­ol­ogy that mea­sures *some* of your web­site vis­i­tors and then projects the total num­ber of unique vis­i­tors from that sam­ple. Depend­ing on which ser­vice you use, this sam­ple can vary dra­mat­i­cally. Recent fig­ures I’ve seen for one of the major audi­ence mea­sure­ment ser­vices sug­gests their US sam­ple is about 120,000 pan­elists. These pan­elists are recruited thru dif­fer­ing strate­gies includ­ing ran­dom dig­i­tal dial­ing (RDD) and/or web-based offers, and the ser­vices work very hard to ensure that these sam­ples are rep­re­sen­ta­tive of the total Inter­net pop­u­la­tion. In turn, these ser­vices project the total US inter­net pop­u­la­tion based on these pan­elists. For exam­ple, as of May 2006, the audi­ence mea­sure­ment ser­vice using 120,000 US pan­elists pro­jected the active US Inter­net pop­u­la­tion at 172 mil­lion. This means, on aver­age, that 1 pan­elist rep­re­sents 1,400 unique visitors.

Unlike web ana­lyt­ics plat­forms, audi­ence mea­sure­ment ser­vices go to great lengths to mea­sure truly unique vis­i­tors. Par­tic­i­pants on their pan­els are required to use sep­a­rate log-ins so that all activ­ity can be tied to an actual indi­vid­ual. Fur­ther­more, these indi­vid­u­als pro­vide a fair amount of demo­graphic and atti­tu­di­nal data about them­selves, so not only can you view these unique vis­i­tors as dis­tinct peo­ple, but you can also dive quite deep into who they are and what moti­vates them.

For all prac­ti­cal pur­poses, audi­ence mea­sure­ment plat­forms could define a unique vis­i­tor as: an indi­vid­ual per­son par­tic­i­pat­ing in a panel that accesses your website.

In this case, the key pieces here are really that the indi­vid­ual is par­tic­i­pat­ing on the panel, and that the panel is rep­re­sen­ta­tive of your web site audi­ence. So refer­ring back to the ini­tial dis­cus­sion, if 1 of these unique vis­i­tors vis­its your web­site, the audi­ence mea­sure­ment ser­vices should report 1,400 unique vis­i­tors to your website.

But in truth, this isn’t nec­es­sar­ily the case. So let’s take a look at some rea­sons why.

NOT ALL PANELISTS ARE CREATED EQUAL
If you dig into audi­ence mea­sure­ment ser­vices, you’ll notice that their “panel” is actu­ally sev­eral pan­els rolled up into one. The 3 major panel types are at-home, at-work, and uni­ver­sity. These cor­re­spond to the actual phys­i­cal loca­tion of the end-user, who can pre­sum­ably access the Inter­net from one or many of these loca­tions. Now, this may seem noth­ing more than a nui­sance; but in real­ity it’s a major issue that audi­ence mea­sure­ment ser­vices strug­gle with. Why? Because while entic­ing at-home users with free virus pro­tec­tion soft­ware tends to be fairly effec­tive, at-work users don’t really care. And even if they did care, few cor­po­ra­tions are will­ing to par­tic­i­pate in these ser­vices. So recruit­ing an at-work panel is *sig­nif­i­cantly more dif­fi­cult and expen­sive* than recruit­ing an at-home panel. And most of those at-work pan­elists are con­fined to smaller firms with less than 100 employ­ees. Tap­ping into medium sized busi­nesses with over 500 peo­ple is next to impos­si­ble; and the For­tune 2000 is all but a pipe dream.

To illus­trate this point, one audi­ence mea­sure­ment ser­vice I recently worked with has a 10:1 ratio of at-home to at-work users. That’s a stag­ger­ing dif­fer­ence. And what’s more — these audi­ence mea­sure­ment ser­vices report that the at-work audi­ence is much more active than the home audi­ence. For instance, in June 2006, Nielsen NetRat­ings reported that at-work inter­net users aver­aged 17 vis­its per per­son and vis­ited 39 domains. This com­pares to the at-home users that aver­age 10 vis­its per per­son and vis­ited just 25 domains. But per­haps most impor­tant is the time spent per per­son. The at-work inter­net user spent roughly 19 hours online in the aver­age week. The at-home inter­net user, by con­trast, spent just under 9 hours per week. More than 50% less!

So the at-work inter­net users rep­re­sent a crit­i­cal pop­u­la­tion of users; yet most audi­ence mea­sure­ment ser­vices have a very dif­fi­cult time access­ing them. This is where the pro­pri­etary audi­ence mea­sure­ment method­olo­gies really kick in. To com­pen­sate for this mas­sive bias in the data, the audi­ence mea­sure­ment firms “weight” the data. In other words, they give more credit to users on the panel that are under-represented and give less weight to those that are over-represented. This weight­ing is based on their assess­ment of what the “nor­mal” inter­net pop­u­la­tion looks like, which they derive thru pro­pri­etary means but ran­dom dig­i­tal dial­ing is cer­tain one fre­quently used approach.

With audi­ence mea­sure­ment ser­vices, not all pan­elists are cre­ated equal and there­fore not all unique vis­i­tors are cre­ated equal. As we’ve seen, there can be huge dif­fer­ences depend­ing on how under-represented a par­tic­u­lar panel demo­graphic may be such as at-work pro­fes­sion­als in For­tune 2000 companies.

QUANTIFYING THE AUDIENCE MEASUREMENT BIAS
From a mea­sure­ment per­spec­tive, this starts to impact sites that are con­cen­trated among par­tic­u­lar demo­graph­ics and devi­ate from the broad inter­net pop­u­la­tion. For exam­ple, if you are a busi­ness to busi­ness tech­nol­ogy web­site, you’ll prob­a­bly draw very few pan­elists to your site because this tar­get demo­graphic is among the hard­est to recruit for the panel. So the audi­ence mea­sure­ment ser­vices must sig­nif­i­cantly weight these few pan­elists to your site and instead of the ear­lier 1 to 1,400 unique vis­i­tor ratio we dis­cussed, it may be more like 1 to 10,000. That’s right — 1 to 10,000 — if not more.

Now, to illus­trate the impact of that bias, let’s say these audi­ence mea­sure­ment ser­vices report you had 1 mil­lion unique at-work vis­i­tors in June 2006. Assum­ing they all more or less meet the above demo­graphic, that’s really only 100 peo­ple that vis­ited your web­site in June 2006 (with a 1 to 10,000 ratio). Of course, it’s unlikely your web­site attracts one dis­tinct demo­graphic — but that’s beside the point, because most demo­graph­ics you do attract will be under­rep­re­sented as it is. So while your web ana­lyt­ics pack­age is mea­sur­ing roughly 100% of unique cook­ied browsers on a unique PC, your audi­ence mea­sure­ment ser­vice is mea­sur­ing just 100 peo­ple at-work. So which is right? Which should you use? I’ll get to that in a bit…let’s dive a bit deeper first.

This panel bias also means that dur­ing abnor­mal peri­ods of inter­net activ­ity, the audi­ence mea­sure­ment ser­vice will likely report even more volatile unique vis­i­tor fig­ures. These abnor­mal peri­ods may include any strate­gic cam­paigns you run, such as search engine mar­ket­ing, email blasts, hol­i­day pro­mo­tions, etc. It may also include sea­sonal spikes such as polit­i­cal events (i.e. pres­i­den­tial elec­tions), sports events (i.e. World Cup, World Series, Foot­ball, Olympics), finan­cial events (i.e. tax sea­son), or just break­ing news sto­ries (i.e. Hur­ri­cane Kat­rina, Iraq War, etc).

Dur­ing these peri­ods, you may see dra­matic spikes in traf­fic reported by your web ana­lyt­ics plat­form, your com­merce plat­form, your ad server, your server web logs, etc. — but the increase reported by audi­ence mea­sure­ment ser­vices will be a frac­tion of those. This has often been a sub­stan­tial point of frus­tra­tion for inter­net mar­keters, because it’s arguably an inac­cu­rate view of per­for­mance. In fact, I’ve worked with some cus­tomers that show a 40% increase in web site traf­fic dur­ing cam­paign peri­ods, and have their audi­ence mea­sure­ment ser­vices actu­ally report a decrease in over­all traffic!

BIGGER SITES ARE STILL AT RISK
There are some schools of thought that sug­gest the larger web­sites are fairly immune to this bias because they attract a fairly rep­re­sen­ta­tive pop­u­la­tion of users. The argu­ment goes that the more rep­re­sen­ta­tive your pop­u­la­tion, the more closely the panel will tie to your demo­graphic mix.

While this is a plau­si­ble the­ory, I find it sus­pect at best. My main issue is that 30–35% of the inter­net pop­u­la­tion accesses the Inter­net from work. As I men­tioned ear­lier, this extends to the time spent on the web as well — with at-work users spend­ing over 2x as much time online as at-home users. What’s more, the at-work audi­ence has come to rep­re­sent a sig­nif­i­cant por­tion of online com­merce spend and this only con­tin­ues to increase. Com­score has reported that the at-work audi­ence con­tributes 60% of online spend. Yet, this cov­eted demo­graphic often rep­re­sents 10% or less of a given audi­ence mea­sure­ment panel — and even that 10% is skewed to small busi­ness and mom-and-pop shops, rather than cor­po­rate Amer­ica at large. So when mea­sur­ing unique vis­i­tors, most web­sites will be adversely impacted by the inher­ent bias of audi­ence mea­sure­ment ser­vices — irre­spec­tive of their size.

LET’S NOT FORGET INTERNATIONAL VISITORS
I lastly wanted to touch on Inter­na­tional vis­i­tors. Most audi­ence mea­sure­ment ser­vices grew out of the United States, though there have been a few Inter­na­tional ones that achieved a healthy foothold in the mar­ket. In any case, Inter­na­tional pan­els tend to be much smaller and frag­mented than their US coun­ter­parts. To under­score that point, many audi­ence mea­sure­ment firms do not even report at-work activ­ity for Inter­na­tional vis­i­tors. As such, many web­sites that rely on these ser­vices to mea­sure unique vis­i­tor traf­fic are miss­ing a mean­ing­ful por­tion of their audi­ence alto­gether. It’s not under-represented; it’s sim­ply not rep­re­sented at all. Iron­i­cally, this issue often becomes more detri­men­tal as your web­site and busi­ness grows.

For exam­ple, one com­pany I have worked for derives over 15% of their rev­enues from Japan. The por­tion is even greater for their online rev­enues specif­i­cally. Japan’s inter­net pop­u­la­tion has been esti­mated at between 80–90 mil­lion users. At least half of these users are access­ing from work, although a few older stud­ies sug­gested as much as 75% of Japan’s inter­net pop­u­la­tion was at-work. In either case, most audi­ence mea­sure­ment ser­vices today do not cur­rently offer at-work pan­els for Japan. In other words, using such ser­vices to gauge over­all web suc­cess would effec­tively cut your audi­ence in half. Of course, as your pres­ence in this mar­ket grows, the gap only increases.

SO WHAT DOES THIS MEAN FOR YOU?
We cov­ered a lot of ground in this first blog post­ing, so take some time to soak it in and pon­der what you’ve just read. If you use web ana­lyt­ics or busi­ness opti­miza­tion plat­forms along­side panel-based audi­ence mea­sure­ment ser­vices, you’re prob­a­bly a bit frus­trated by all of this. Why can’t I just have one right answer, instead of two answers that are arguably both wrong? Why doesn’t some­one offer the best of both worlds? Hasn’t some­one solved this prob­lem yet?

Indeed, these are all fair ques­tions. The key is that each respec­tive mea­sure­ment approach is actu­ally quite valu­able — if you know how to apply them. Web ana­lyt­ics or busi­ness opti­miza­tion can be lever­aged to drive sub­stan­tial gains in your top and bottom-line. Audi­ence mea­sure­ment ser­vices can also be lever­age to inform sig­nif­i­cant strate­gic deci­sions. But to get there, you need to first stop com­par­ing num­bers from each sys­tem. Once you’ve got­ten com­fort­able with that con­cept, the pos­si­bil­i­ties for busi­ness improve­ment are vir­tu­ally limitless.

In my next blog post­ing, I’ll share my thoughts on how you might con­sider using both mea­sure­ment plat­forms to achieve greater busi­ness suc­cess. Until then, I wel­come your feed­back and comments!

0 comments