In my last post, I dis­cussed audi­ence mea­sure­ment — unique vis­i­tors, page views, time spent on site, and impres­sions — and why I believe time spent on site is not, con­trary to what some are say­ing in the trade press, the best met­ric for mea­sure­ment. Today, more details on why page views are not nec­es­sar­ily the best mea­sure of vis­i­tor engagement…

As with other adver­tis­ing medi­ums, the online audi­ence mea­sure­ment indus­try was born out of the need to pro­vide pub­lish­ers with a com­mon cur­rency by which they could mar­ket and qual­ify their sites to prospec­tive advertisers.

If I’m an adver­tiser look­ing to reach 1 mil­lion peo­ple with a new movie pro­mo­tion, how would I know which media sites I should adver­tise on? If I had a com­mon met­ric — or a com­mon set of met­rics — I could quickly scan that list to find sites that reach 1 mil­lion peo­ple, and then buy that space to reach my audience.

For such sim­pli­fied audi­ence mea­sure­ment like this, Nielsen is the de facto stan­dard in the offline world, and at the out­set of the Inter­net, there was no online equivalent.

In the late 1990’s, audi­ence mea­sure­ment firms sought to be this com­mon cur­rency, offer­ing up met­rics like unique vis­i­tors, page views, and time-spent-on-site. Nielsen was one of the first to throw its hat in the ring. They cre­ated a ser­vice that pro­jected these met­rics for major Inter­net sites, based on a panel they main­tained of sev­eral thou­sands users.

This is nearly iden­ti­cal to their offline approach, and why not: if it worked in the offline world, why not give it a shot online? The chal­lenge is that this panel-based approach is eas­ily skewed and only use­ful at a very high-level. As I’ve talked about in the past, that’s because the Inter­net offers the poten­tial to suc­cess­fully reach peo­ple in extremely nar­row niches of inter­est. If you’re a knit­ter who also likes to quilt but who hates to cro­chet, there’s prob­a­bly a web­site for you and oth­ers with the same likes and dis­likes. On the other hand, it is highly unlikely that, even with its panel of thou­sands, an audi­ence mea­sure­ment panel will have many knit­ting, quilt­ing, crochet-haters on its panel.

Audi­ence mea­sure­ment firms will likely then strug­gle to mea­sure the niche-y craft site, when in real­ity, that site may see tens of thou­sands of vis­i­tors per month. A yarn com­pany look­ing for places to adver­tise, but who goes only by panel responses, may miss out on the site com­pletely, never know­ing there was a small but impor­tant group of crafts enthu­si­asts poten­tially eager to see the yarn company’s ads. And as many folks know, loyal cus­tomers can be 7x more valu­able than new cus­tomers, so tap­ping into this niche cus­tomer seg­ment is critical.

Along these same lines, tar­geted direct mar­ket­ing ini­tia­tives like email cam­paigns, paid search, new microsites, etc., can also be under­stated by such panel ser­vices. Again, those ini­tia­tives are likely to hit only a hand­ful of the pan­elists, and a “hand­ful” is gen­er­ally viewed as not being sta­tis­ti­cally sig­nif­i­cant enough to sur­face as a mean­ing­ful trend or change.

Sim­i­larly, when sites add new con­tent — new arti­cles, spe­cial edi­tions, etc. — these can be under­stated or unde­tected. By how much? There’s no way to tell for sure, unless you use web ana­lyt­ics, which is arguably the most accu­rate way to mea­sure the suc­cess of these initiatives.

Still in doubt? Run a sim­ple test. If you’re a retailer, look at how many orders you have on a given day as reported by your com­merce engine. Now check your web ana­lyt­ics plat­form. The orders, gen­er­ally speak­ing, should be within 2–3% — if not per­fectly in line. Now, check with an audi­ence mea­sure­ment firm – what are they report­ing for the day? I’ve done this mul­ti­ple times and never seen any­thing close to accu­rate. If you’re not a retailer, pick some­thing else – like leads, appli­ca­tions, etc – that you can val­i­date not only with web ana­lyt­ics but a back-end sys­tem. The key to this exer­cise is tri­an­gu­la­tion so you need at least one more data source beyond your web ana­lyt­ics and audi­ence mea­sure­ment services.

Of course, site-side ana­lyt­ics has his­tor­i­cally offered very lit­tle to adver­tis­ers in eval­u­at­ing com­pet­ing sites, so I read­ily acknowl­edge that audi­ence mea­sure­ment can be a use­ful proxy for com­par­a­tive traf­fic lev­els (as I’ve writ­ten about in the past.)

Still, in the late 1990s, when audi­ence mea­sure­ment firms intro­duced these pan­els, adver­tis­ers were under­stand­ably excited, because at least they could com­pare one site to another with the same met­rics. In fact, for some time, ven­ture cap­i­tal­ists and invest­ment bankers often used these same ser­vices to esti­mate val­u­a­tions for pre-IPO inter­net com­pa­nies, using unique vis­i­tors as the mea­sure of “eye­balls” the site could pre­sum­ably mon­e­tize into pay­ing cus­tomers some day.

Around the same time, page views also came to be viewed as a mea­sure of engage­ment. Folks began to real­ize that not all unique vis­i­tors are cre­ated equal: two sites that each have 1 mil­lion vis­i­tors can be very dif­fer­ent from each other in terms of reach, if most of the vis­i­tors to one of the sites come to the home page and then leave imme­di­ately, while vis­i­tors to the other site stay and browse.

Page views, then, became the check and bal­ance against unique vis­i­tors, and ide­ally the two taken together could pro­vide a rounded assess­ment of site engage­ment and rev­enue potential.

The chal­lenge with page views is that they are actu­ally not stan­dard­ized. Nielsen and other audi­ence mea­sure­ment firms could con­trol unique vis­i­tors because they man­aged the pan­els them­selves. They paid or oth­er­wise com­pen­sated each mem­ber of the panel so that unique­ness was fairly well pre­served.

But audi­ence mea­sure­ment firms do not and can­not con­trol page views because they source from the sites them­selves. Pages come in all dif­fer­ent shapes and sizes, some with dynamic con­tent and some that are com­pletely sta­tic. Not all page views are cre­ated equal – and audi­ence mea­sure­ment firms are faced with the impos­si­ble task of try­ing to cre­ate a com­mon stan­dard. And let’s pre­tend for a sec­ond that this was achievable…that audi­ence mea­sure­ment firms had picked apart every web page from every site, and clas­si­fied it as a page view. Well, Web con­tent can change mul­ti­ple times per day per site so while the utopian stan­dard could have the­o­ret­i­cally been accu­rate, it would have quickly become inac­cu­rate as con­tent and lay­out changed.

For exam­ple, there is the stan­dard HTML page that we all know. That’s fairly easy to stan­dard­ize across sites.

But then there are gen­er­ated pages with dynamic URLs such as retail web­sites that cre­ate new URLs on the fly for each prod­uct. What do you do with that? On top of that, you have dynamic pages that do not change the URL at all (see my exam­ple about the GAP in my pre­vi­ous post.

And stream­ing media and wid­gets are not even pages — they are com­plete expe­ri­ences in and of themselves.

As the Web has evolved, these “non-traditional” pages have become increas­ingly preva­lent, because in many cases, they pro­vide a supe­rior cus­tomer experience.

So while page views emerged as an early mea­sure of engage­ment, it really was never fair to com­pare it across sites — whether they were tracked by a panel or otherwise.

In my next post, I’ll dis­cuss audi­ence mea­sure­ment firms’ “new” met­ric, time-spent-on-site.

2 comments
Susan
Susan

I still like to measure pageviews per visitor, and i'm curious as to the industry average? I have never been able to find an accurate number. I guess by reading this article above, you really can't compare that either?

Rob Blakeley
Rob Blakeley

Some thoughts: If advertisers want to pay for time-spent, they will get it or they may go elsewhere. That aside, what's the goal of engagement, time-spent , or any other metric or combination of metrics ? Follow the money. If the metric trend line matches the profit trend line, then you have validated the metric. In fact, there is a good chance that the metric would be predictive. If not, stop using it. Who's money? If you are a retailer, it's your money. If you are an advertiser, then it's sort of your money. They pay you to drive their profit. Demonstrating that with your metric would require more cooperation and risk than most companies are willing to undertake.