Visitor Engagement: Time for a reality check?

Analytics · By On · 13 Comments

One of my con­sul­tants caught me in the hall last week and asked what I thought about vis­i­tor engagement. A cus­tomer of his devel­oped a com­plex for­mula — a mashup of met­rics — to mea­sure “Vis­i­tor Engage­ment” on his site. The idea is that this will fea­ture promi­nently in exec­u­tive dash­boards.  If the num­ber goes up, great.  If it goes down, they’ll take action to rec­tify the situation.

This sounds like the answer to our engage­ment prayers: one met­ric to mea­sure all.  It’s the Esperanto of engage­ment, a com­mon lan­guage by which we can under­stand the customer.

Unfor­tu­nately, I think it’s a ter­ri­ble idea. In many ways, it’s the antithe­sis of all that mea­sure­ment stands for in my mind.  Why?  Let me explain.

The Basic Premise of Mea­sure­ment
The basic premise of mea­sure­ment is that you want to mea­sure some­thing so you can improve it, if necessary. If my body/fat ratio is out of whack, I’ll work out and eat bet­ter to bring it back in line. If my con­ver­sion rate is lower than my his­tor­i­cal aver­age, I’ll try to improve it.  If cam­paign response is weak, I’ll look at some fresh creative.

It’s pretty sim­ple – col­lect data, ana­lyze, improve.  I love this because of its sim­plic­ity and objectivity. In my early days of ana­lyt­ics, I spent count­less hours watch­ing as exec­u­tives argued emo­tions instead of facts.  And, unfor­tu­nately, back in the late 90s, ana­lyt­ics were hardly robust enough to con­fi­dently argue in favor of either side.  Gen­er­ally the per­son with the big­ger title won the argu­ment and their rec­om­men­da­tions were put into place.

But now, ana­lyt­ics are far more robust (when imple­mented and man­aged prop­erly), and we live in a won­der­ful world of objec­tiv­ity (for the most part).

So what hap­pens when you start com­bin­ing met­rics into uber-formulas like Vis­i­tor Engage­ment?  That model breaks, because you intro­duce a level of abstrac­tion on the data. You “dumb it down,” intro­duc­ing bias and subjectivity.

Break­ing the model: why uber met­rics don’t work
Let’s say ‘engage­ment’ is clas­si­cally defined as leads/visits on the site.  That’s an objec­tive mea­sure of how a visitor’s expe­ri­ence is lead­ing to a pos­i­tive out­come for both par­ties.  In other words, it’s a mea­sure of how engaged the vis­i­tor has become in his rela­tion­ship with a com­pany, and it demon­strates a strength­en­ing rela­tion­ship – all good things in the world of cus­tomer management.

Now let’s say you cre­ate an engage­ment mashup.  The mashup includes vis­i­tors that have returned “often” to the site as one met­ric, when they view “impor­tant” con­tent as another met­ric, and, just for good, mea­sure, we’ll include vis­i­tors that spent a “long” time on the site as the final metric.

That’s just three met­rics; it can’t be that biased, right?  You bet it can.

First, what kind of return fre­quency is “often” — two vis­its?  Four?  Six?  That’s sub­jec­tive.  What is “impor­tant” con­tent?  The home page?  An arti­cle?  A sup­port document? Subjective again.  And what is a “long” time on site — 5 min­utes, 10 min­utes?  Per­haps “long” means any visit that exceeds the aver­age for the site that week?

You can see how quickly this becomes totally sub­jec­tive.  Because of its sub­jec­tiv­ity, it has become totally worth­less.  You have intro­duced mas­sive bias with­out com­ing up with a met­ric that allows you to make decisions.

Let’s say this for­mula yields a Vis­i­tor Engage­ment “Score” of 40 for last month.  This month, the same for­mula pro­duces a score of 30.  That’s a pretty dire sit­u­a­tion — but what do you do about it?  How can your exec­u­tive team act on that num­ber?  They can’t!  Your best hope is to begin dis­sect­ing the Vis­i­tor Engage­ment score to its fun­da­men­tal met­rics and fig­ure out which one is respon­si­ble for the decrease.

For exam­ple, sup­pose return fre­quency was flat, vis­its to impor­tant con­tent sky­rock­eted, but time on site fell through the floor.  You’ll prob­a­bly want to focus on time spent on site, and see if you can improve that.  But if your pri­mary KPI of leads/visits has increased (i.e. your con­ver­sion), maybe you’ve actu­ally done a really good thing and you should leave it alone. You’ve cre­ated a more fric­tion­less expe­ri­ence, and the declin­ing Vis­i­tor Engage­ment score sup­ports this.

At this point, you’ll face the unde­sir­able task of con­vinc­ing your execs that the Vis­i­tor Engage­ment met­ric, which you fought so hard to social­ize and adopt, should actu­ally decline.

But WAIT! Not all uber met­rics are bad
So I think you get the point.  Vis­i­tor engage­ment for­mu­las are largely another fad, just like para­chute pants and the Hol­ly­wood diet.  It’s a mea­sure some con­sul­tants and ven­dors can pitch like snake oil.

But, that is not to say that uber met­rics are com­pletely worth­less.  In select cases, you can actu­ally lever­age uber for­mu­las to make very use­ful decisions.

Uber met­rics that are purely objec­tive can hold value to an orga­ni­za­tion.  Per­haps one of the great­est is RFM – Recency Fre­quency Mon­e­tary.  In that case, you’re deal­ing with an (almost) entirely objec­tive uber met­ric.  For those not as famil­iar with RFM, it’s a clas­sic cus­tomer seg­men­ta­tion tech­nique that essen­tially calls for you to score your cus­tomers based on their ‘rel­a­tive’ rank to one another along three pri­mary metrics.

You then roll up these scores to arrive at an uber score, and iden­tify your best (high­est scor­ing) and worst (low­est scor­ing) cus­tomers.  Action you can take from learn­ings gleaned from this analy­sis are too numer­ous to name.  It’s actu­ally a lot of fun to do these kinds of models. But even in this case, sub­jec­tively can often enter the picture.

For exam­ple, the time­line over which you ana­lyze cus­tomer data is one of the prin­ci­pal points of sub­jec­tiv­ity.  In the RFM model above, do you ana­lyze behav­ior over 1 month, 6 months, 1 year or 6 years?   Maybe you just take as much data as you can find and mix it all together and hope for the best.  In turn, once you com­plete your RFM seg­men­ta­tion, what time period do you com­pare it to?  Weeks?  Months? Years?  Again, subjective.

RFM has a long his­tory of being valu­able – so again, I’m not throw­ing uber met­rics under the bus entirely.  Still, I wouldn’t waste your time with most of them.  There are so many oppor­tu­ni­ties for opti­miza­tion based on pri­mary key per­for­mance indi­ca­tors like con­ver­sion that you can keep your entire team busy for years.

Don’t try to build a bet­ter mouse trap, when you’re not tak­ing advan­tage of the one you’ve got today.

So, those are my thoughts.  As always, I wel­come your ideas and feedback.

Tagged with →  
  • http://blog.jimnovo.com/ Jim Novo

    Hi Matt -

    I’m with you on this in terms of the prac­ti­cal­ity of explain­ing and act­ing on these com­plex mod­els. While the per­son who con­tructs the model may be able to “see it in their head”, it’s tough to press for­ward from there in the org.

    And I would also argue that RFM was the very first engage­ment model and still a wildly suc­cess­ful one at that, though for the pur­poses of the web we might drop the “M” so we get a model that applies out­side of commerce.

    With RF, now you’re down to two ques­tions: How many times did they act (F) and how long ago was the last action ®? Sim­ple to explain to just about any­one and the R com­po­nent is strongly pre­dic­tive of like­li­hood to act in the Future, which gets us into pre­dict­ing the Poten­tial Value of a Vis­i­tor / Customer.

    In the long run, I think peo­ple will find that mea­sur­ing dis-Engagement (Recency) is much more impor­tant than mea­sur­ing Engage­ment, because it tells you the high­est ROI point to act.

    Peo­ple who are Engaged don’t need much Mar­ket­ing. Peo­ple who have dis-Engaged are a lost cause. Peo­ple who are in the process of dis-Engaging, if you can catch them, is where you can make a lot of money.

  • http://blackbeak.com Steve Jack­son

    Hi Matt,

    I had to chime in here because I am con­sis­tently see­ing this debate appear again. This is a recent excerpt I posted on Web­An­a­lyt­ics­De­mys­ti­fieds’ Future of blog talk­ing about the same thing;

    “It goes back to a point I first read about ages ago on Eric’s blog (late 2006) and then when Eric pub­lished his engage­ment for­mula became the leg­endary “engage­ment” debate I’m sure we all remem­ber, on Occam’s Razor, Jims site, my site and a bunch of oth­ers. It got quite heated at times as it should. Pas­sions were ignited and peo­ple were draw­ing lines in the sand. At the time I took a step back and looked at what we all were say­ing and came to the con­clu­sion we were largely debat­ing seman­tics though we all agreed on some things.

    What we pretty much all agree on is that the more ways we have of iden­ti­fy­ing ways to get more cus­tomers and take actions on our met­rics the bet­ter. Note the “take action” part. I think most of us agree if you can’t act on the KPI then why use it.

    We all agree that we have to use quan­ti­ta­tive (click­stream), qual­i­ta­tive (voice of cus­tomer, atti­tu­di­nal) and com­pet­i­tive (com­par­i­son) data to drive the best insights. Learn­ing to com­bine these data sources is the way the indus­try will move. ”

    I think you would agree if you can take action on a met­ric then you can use it judg­ing by the points you make above.

    In the “leg­endary debate” Avinash argued the same point as you’re mak­ing and I’m not dis­agree­ing with either per­spec­tive here. The whole point of every­thing is con­ver­sion eventually.

    The prob­lem then was one guy was dis­cussing engage­ment in the con­text of RF mod­els. One was dis­cussing engage­ment in terms of bounce rate, one was dis­cussing engage­ment in terms of scor­ing actions and I was defin­ing engage­ment seg­ments and every­one else was throw­ing their 2 cents into the mix till you had a whole big bag of ideas labeled as engagement.

    I have used Omni­tures (and other) tools suc­cess­fully to save clients mil­lions of euros using noth­ing but engage­ment seg­ments as I define them. No con­ver­sion was made ini­tially but the like­li­hood to con­vert later (mea­sured via RF) was much higher and by using engage­ment indexes to opti­mize key­word spend and ban­ner place­ments we saved clients a lot of money.

    As a con­sul­tant using your tools it’s our respon­si­bil­ity to make or save our clients money. I often find that in order to do that I have to do a lot more than sim­ply mea­sure con­ver­sion fun­nels because all that does is show low hang­ing fruit which will only give you so much (and only works for some sites).

    The puz­zle I am being asked to solve more and more is this.

    How come multi-channel cam­paigns for the past 12 months have shown only aver­age or unre­mark­able con­ver­sion results and yet our prof­its are still ris­ing excep­tion­ally? Which part of our ad money are we wast­ing? Some­thing is work­ing but which is it?

    In the above sit­u­a­tion con­ver­sion rate alone is worth­less. Much like engage­ment index alone, or bounce rate alone or page views alone.

    The point I think folks might be miss­ing is that it’s not about “one met­ric to mea­sure all” it’s about tak­ing var­i­ous met­rics in con­text to each other and know­ing how to inter­pret and act on those metrics.

    Best,
    Steve.

  • Nathan Janitz

    While in prin­ci­ple I dis­agree with Matt, I see where he is com­ing from with the “engage­ment” met­rics being very sub­jec­tive. But as Steve pointed out, “As a con­sul­tant using your tools it’s our respon­si­bil­ity to make or save our clients money. I often find that in order to do that I have to do a lot more than sim­ply mea­sure con­ver­sion fun­nels because all that does is show low hang­ing fruit which will only give you so much (and only works for some sites).”

    One can test until they are blue in the face, but at some point we have to answer the ques­tions: “why is this not work­ing”. You can­not do this with­out mea­sur­ing some form of engage­ment. Lead­ers in this indus­try don’t become lead­ers by just giv­ing up after the last test; they fig­ure out a way to answer ques­tions. They try to under­stand the why.

    While I agree with Matt’s state­ment about build­ing a bet­ter mouse trap, you also can’t catch a mouse with­out look­ing at the right information….all of the right infor­ma­tion. While 1 met­ric can deter­mine absolute suc­cess (how does this make me money), it can never answer the ques­tions “why”, “how”, or “so what.”

    As Steve said, look­ing at the prob­lem through on KPI is worth­less. I would also add look­ing at them with­out the right con­text is also worth­less. Is 5 sec­onds or 5 min­utes the right time-on-page? The answer com­pletely depends on the con­text. 5 min­utes on a check­out page or spend­ing 5 sec­onds watch­ing a 6 minute video can both give insight into how a per­son engages with the content/website.

    Some­day the dis­cus­sion of what defines “engage­ment” will become a thing of the past and be replaced with the next new idea. The meth­ods them­selves will stay and evolve as we test, eval­u­ate, and test again (basi­cally how web ana­lyt­ics itself has evolved). Again, at some point logic must take part and real­ize that by nature “engage­ment” is some­what sub­jec­tive, but can be qual­i­fied when put into the right con­text. If web ana­lyt­ics were as sim­ple as just con­ver­sions, most web ana­lyt­ics plat­forms would be a sin­gle page with a pass/fail reading.

    The world is not always as black or white as we might like. It is our respon­si­bil­ity to make sure that we max­i­mize our client’s money. As thought lead­ers in the indus­try, it is also our respon­si­bil­ity to make sure that we are look­ing at new ways of explain­ing what is hap­pen­ing on the web­site. The most recent way of doing that is by try­ing to define and mea­sure engage­ment. It’s not a fad; it is an evo­lu­tion of the indus­try… the fad is more likely the way we view/measure engagement.

  • http://www.jdk.de CMS Spe­cial­ist

    Do you have any expe­ri­ences on cre­at­ing real­time feed­back on the scores you are calculating?

  • Dr. James Joseph Geertz

    Matt,

    I would be curi­ous as to who the com­pany was that you found pro­duc­ing “vis­i­tor engage­ment”, as well as which met­rics, if any, they were employ­ing sub­jec­tively and which met­rics, if any, they were employ­ing objec­tively. Your post­ing doesn’t make it clear that you actu­ally per­formed any seri­ous analy­sis of the for­mula before dis­miss­ing it as an inac­cu­rate uber for­mula. Rather you jumped from call­ing the for­mula a mashup of met­rics to berat­ing a hypo­thet­i­cal for­mula of met­rics mashups.

    There is a won­der­ful world of math­mat­ics out there. It is short-sighted to believe that, with all the data that can be col­lected by site hosts, the end all be all of web ana­lytic tools is going to be a divi­sion prob­lem with two count­ing sta­tis­tics (leads/visits). If you are not going to throw the for­mula down on the mat, dis­play it in the post, and point out the flaws, it is hard to under­stand how your view is not sim­ply a heavy-handed dis­missal of all non-traditional statistics.

    Hope­fully Omni­ture is not nearly so close minded.

    Regards,
    James

  • http://blogs.omniture.com/author/mbelkin/ Matt Belkin

    James,

    Thanks for your com­ments. Did you hap­pen to read the last para­graph of my blogs post? I actu­ally talk about RFM and the value of that approach, which ties pre­cisely into your thoughts. If have you haven’t done so, please read that part of my post and let me know if you have questions.

    Thx, Matt

  • http://www.shoutdomains.com David Scov­ille

    Thanks for the infor­ma­tion. I think uber met­ric num­bers make it easy to share quick infor­ma­tion with exec­u­tives or clients–to show them basi­cally where they were then, and where they are now. I do agree with you on the sub­jec­tiv­ity of uber metrics…and I’m still try­ing to swal­low RFM.

  • Amadeus Fagereng

    I am all new to this, and it may be a stu­pid com­ment, but I will post it anyway.

    I do not see the prob­lem with sub­jec­tive para­me­ters, as long as the for­mula is cus­tomized towards the spe­cific company/website. After work­ing with the company/website for a while you should know what the fre­quency “often” would be for your site.

    By using your pre­vi­ous analy­sis you will be able to pro­duce an over­all mea­sure­ment of your sites per­for­mance with­out hav­ing to per­form­ing repet­i­tive man­ual and sub­jec­tive analy­sis. Also, the time it takes to come up with a mea­sure­ment is cru­cial, and by hav­ing such an indi­ca­tor you can deliver live performance.

    Of course, this is not the same as say­ing the for­mula would replace other meth­ods, but I believe it would be a good indi­ca­tor on when you want to look into other meth­ods. The num­ber would serve as and indi­ca­tor of change in performance.

    I believe it is cru­cial to have such live (or close to) mea­sure­ments that actu­ally take into account sub­jec­tive parameters/metrics of your website.

  • http://www.thelostagency.com David

    I had an inter­est­ing dis­cus­sion today with a client about a project to improve their con­ver­sion rates where they wanted imme­di­ate engage­ment. Two points about this one is that the client is hav­ing to cal­cu­late the ROI of the engage­ment on the value of the client now, as the prod­uct is sub­scrip­tion soft­ware it too dif­fi­cult to cal­cu­late the life time value of the client as you can­not pre­dict how long a client will con­tinue to use your prod­uct. If the prod­uct is a yearly sub­scrip­tion it can pro­vide feed­back for the fol­low­ing year as to those who came from the cam­paign who con­tin­ued with sub­scrip­tion. The prob­lem is hav­ing a long enough view of your sales and how mak­ing deci­sions that will affect your poten­tial sales for this year now with­out this data.

    Another client can use the vis­i­tor engage­ment around the num­ber of return­ing vis­i­tors as a posis­tive behav­iour and how this relates to who order an infor­ma­tion pack.

    Can return­ing vis­i­tors be a unchanged vis­i­tor engage­ment met­ric that is con­sis­tent between industries?

  • Cory Hen­drick­son

    I have had the oppor­tu­nity to work with both engage­ment met­rics and RFM. While I agree that RFM is a valu­able mea­sure — the world of loy­alty is solely based on this met­ric — it has not adapted to a dig­i­tal world. The chal­lenge here is in cus­tomer seg­men­ta­tion. There is lit­tle flex­i­bil­ity in an RFM model to account for exter­nal behav­ior. We assume that web prop­er­ties remain islands com­pet­ing (some­what) equally for time.

    Today, this is not true. Con­ver­sion does not always hap­pen at a sin­gle domain. Con­ver­sion is a prod­uct of both explicit search and implicit dis­cov­ery. Where RFM is domain and cus­tomer depen­dent, engage­ment met­rics aim to pro­vide alge­braic func­tions to ana­lyze strengths and weak­nesses against larger (and often uniden­ti­fied) segments.

    Suc­cess­ful engage­ment met­rics do not mea­sure the engage­ment for one cus­tomer like defined here. A suc­cess­ful engage­ment met­ric is com­prised of var­i­ous inputs to help ana­lysts iden­tify strong chan­nels. Per­haps some refer­ring sites pro­vide bet­ter con­ver­sion than oth­ers. Maybe time spent on a site is the great­est indi­ca­tor of suc­cess. Engage­ment pro­vides for loose inter­pre­ta­tion of both domain depen­dent and domain inde­pen­dent cus­tomer events to help iden­tify the proper met­rics that are most impor­tant for any given business.

    While cer­tainly not appro­pri­ate for an executive’s daily dash­board report, they are the only cur­rent solu­tion I have seen that answers the ques­tion — Why do we care about this spe­cific met­ric more than the others?

  • Matthew Pax­man

    Matt,

    It seems that the whole issue is not really a com­bi­na­tion of met­rics, in and of itself, but the built-in assump­tions that bias the sys­tem. What we need to do is tell the sys­tem what means end-of-the-day con­ver­sion to us, and then let IT tell us how our cus­tomers are get­ting there (and where we have cre­ated dead-ends and road­blocks, killing con­ver­sion, or really turn­ing on the green light to let the good times roll). After all, the whole point of ana­lyt­ics is to give your­self insight to make deci­sions, not make just make deci­sions and then look at pretty num­bers on a page afterword.

    Also, the other prob­lem encoun­tered is how vague the out­come has become. Does Vis­i­tor Engage­ment, or your uber-metric d’jour, even mean your objec­tive as an orga­ni­za­tion was reached? not only is the out­come sub­jec­tive, but it is wrapped-up, mul­ti­plied, nor­mal­ized, starched and ironed to mean zilch in and of itself. We need data val­ues that speak to us, not buzz-worded cure-all won­der metrics.

    The take away? Don’t tell the sys­tem what to give you back, rather let the sys­tem tell you what engaged and non-engaged Vis­i­tors and Cus­tomers are doing and use those val­ues in your mea­sure­ment. And above all, make sure to have a run­ning analy­sis of what the ana­lyt­ics are telling you so you can adjust what val­ues you have pulled into your for­mula and thus be flex­i­ble so far as the dynamic of your Vis­i­tors and Cus­tomers defines.

  • http://www.manthan.com Man­than

    RFM model has indeed come a long way to mea­sure cus­tomer loy­alty and build more tar­geted loy­alty mar­ket­ing pro­grams. A cus­tomer who has pur­chased recently and fre­quently and cre­ated a high mon­e­tary value through these pur­chases is much more likely to pur­chase again. Such cus­tomers are called high RFM cus­tomers. On the other hand, cus­tomers who have not pur­chased in a long time tend to be com­par­a­tively less inter­ested in the store/brand. Adding the counts for recency, fre­quency, and mon­e­tary value presents a good indi­ca­tor of inter­est in the store/brand at the cus­tomer level. This is valu­able infor­ma­tion for a retail busi­ness to have. For more details on using RFM model, visit: http://​thoughts​.man​thansys​tems​.com/​r​f​m​m​o​d​e​l​_​b​u​s​i​n​e​s​s​_​a​n​a​l​y​t​i​c​s​.​php

  • http://www.galerie-lounge.de galerie-lounge

    I think uber met­ric num­bers make it easy to share quick information