Since I wrote about bench­mark­ing two weeks ago, I’ve received many fol­low up emails. Many of you said “Amen! Thanks for dis­pelling the myths around web ana­lyt­ics bench­mark­ing.” How­ever, some of you pushed back, say­ing that “con­ver­sion bench­marks are use­ful if you sim­ply weigh them appro­pri­ately.” And truth be told, I do not dis­agree with the sec­ond point. Bench­marks can be use­ful if you weigh them appro­pri­ately – of course, when com­pared to site-centric data, I give bench­marks very lit­tle weight.

To illus­trate why bench­marks are often mis­lead­ing at best, take a look at the Fireclick Index (http://​index​.fireclick​.com/). I’ve focused on this index because it’s pub­licly avail­able and every­one can read­ily access it.

In look­ing at the index, you’ll first notice the Global Con­ver­sion rate is down 17%. Is that con­ver­sion for all web­sites, i.e. lead gen­er­a­tion, media, etc? Or is that just retail? Based on the pres­ence of shop­ping cart aban­don­ment fur­ther down the page, I’d guess this index is only focused on retailers.

So let’s assume I’m a retailer and there’s some rel­e­vance here to my busi­ness. Global con­ver­sion is down 17%. What does that tell me? That a com­peti­tor screwed up their check­out process? That a competitor’s site is worse than mine? That the indus­try in gen­eral just wrapped up a major tradeshow? That someone’s home­page pro­mo­tion failed? Should I invest mil­lions into a land grab cam­paign to take advan­tage of this brief win­dow of mar­ket oppor­tu­nity? Let’s say my con­ver­sion went up 15%, but I didn’t do any­thing dif­fer­ently on my site. What does this tell me? What deci­sion am I going to make? This data, while inter­est­ing, doesn’t sup­port any strate­gic deci­sion I’d want to make with confidence.

Mov­ing down the list, I see con­ver­sion for key­words is 0.90%. Inter­est­ing, except I bid for key­words based on ROAS, not on con­ver­sion. If my key­words were con­vert­ing at 10%, does that mean I should buy more because I’m higher than the index? Should I bid higher? No, of course not. Because any search mar­keter worth their salt knows a 10% con­ver­sion rate does not tell you how much profit these key­words are generating.

I’m still look­ing for some­thing action­able – turn­ing now to email. Inter­est­ing to see the 1.5% con­ver­sion. Are these acqui­si­tion cam­paigns? Are these third-party lists? HTML email, or just text? Are these auto-bounce backs? Do they include pro­mo­tions with pric­ing dis­counts, or just brand emails? Do they also include newslet­ters? Well, with­out these details, it’s pretty use­less com­par­ing my email to this index.

Aver­age Con­nec­tion Speed dropped 1%. Hmm, what do I do about this? I can com­pare this to my site con­nec­tion speed. My con­nec­tion speed decreased 10%. Well, since I’m 9 per­cent­age points below the aver­age, should I start buy­ing more server hard­ware? Increase the pipe? Out­source to Aka­mai? Again, this bench­mark data sim­ply doesn’t deliver action intelligence.

So that’s my take on bench­marks. Despite claims to the con­trary, I have yet to see a per­sua­sive argu­ment for tak­ing strate­gic action from bench­mark data.