One of my con­sul­tants caught me in the hall last week and asked what I thought about vis­i­tor engagement. A cus­tomer of his devel­oped a com­plex for­mula — a mashup of met­rics — to mea­sure “Vis­i­tor Engage­ment” on his site. The idea is that this will fea­ture promi­nently in exec­u­tive dash­boards.  If the num­ber goes up, great.  If it goes down, they’ll take action to rec­tify the situation.

This sounds like the answer to our engage­ment prayers: one met­ric to mea­sure all.  It’s the Esperanto of engage­ment, a com­mon lan­guage by which we can under­stand the customer.

Unfor­tu­nately, I think it’s a ter­ri­ble idea. In many ways, it’s the antithe­sis of all that mea­sure­ment stands for in my mind.  Why?  Let me explain.

The Basic Premise of Mea­sure­ment
The basic premise of mea­sure­ment is that you want to mea­sure some­thing so you can improve it, if necessary. If my body/fat ratio is out of whack, I’ll work out and eat bet­ter to bring it back in line. If my con­ver­sion rate is lower than my his­tor­i­cal aver­age, I’ll try to improve it.  If cam­paign response is weak, I’ll look at some fresh creative.

It’s pretty sim­ple – col­lect data, ana­lyze, improve.  I love this because of its sim­plic­ity and objectivity. In my early days of ana­lyt­ics, I spent count­less hours watch­ing as exec­u­tives argued emo­tions instead of facts.  And, unfor­tu­nately, back in the late 90s, ana­lyt­ics were hardly robust enough to con­fi­dently argue in favor of either side.  Gen­er­ally the per­son with the big­ger title won the argu­ment and their rec­om­men­da­tions were put into place.

But now, ana­lyt­ics are far more robust (when imple­mented and man­aged prop­erly), and we live in a won­der­ful world of objec­tiv­ity (for the most part).

So what hap­pens when you start com­bin­ing met­rics into uber-formulas like Vis­i­tor Engage­ment?  That model breaks, because you intro­duce a level of abstrac­tion on the data. You “dumb it down,” intro­duc­ing bias and subjectivity.

Break­ing the model: why uber met­rics don’t work
Let’s say ‘engage­ment’ is clas­si­cally defined as leads/visits on the site.  That’s an objec­tive mea­sure of how a visitor’s expe­ri­ence is lead­ing to a pos­i­tive out­come for both par­ties.  In other words, it’s a mea­sure of how engaged the vis­i­tor has become in his rela­tion­ship with a com­pany, and it demon­strates a strength­en­ing rela­tion­ship – all good things in the world of cus­tomer management.

Now let’s say you cre­ate an engage­ment mashup.  The mashup includes vis­i­tors that have returned “often” to the site as one met­ric, when they view “impor­tant” con­tent as another met­ric, and, just for good, mea­sure, we’ll include vis­i­tors that spent a “long” time on the site as the final metric.

That’s just three met­rics; it can’t be that biased, right?  You bet it can.

First, what kind of return fre­quency is “often” — two vis­its?  Four?  Six?  That’s sub­jec­tive.  What is “impor­tant” con­tent?  The home page?  An arti­cle?  A sup­port document? Subjective again.  And what is a “long” time on site — 5 min­utes, 10 min­utes?  Per­haps “long” means any visit that exceeds the aver­age for the site that week?

You can see how quickly this becomes totally sub­jec­tive.  Because of its sub­jec­tiv­ity, it has become totally worth­less.  You have intro­duced mas­sive bias with­out com­ing up with a met­ric that allows you to make decisions.

Let’s say this for­mula yields a Vis­i­tor Engage­ment “Score” of 40 for last month.  This month, the same for­mula pro­duces a score of 30.  That’s a pretty dire sit­u­a­tion — but what do you do about it?  How can your exec­u­tive team act on that num­ber?  They can’t!  Your best hope is to begin dis­sect­ing the Vis­i­tor Engage­ment score to its fun­da­men­tal met­rics and fig­ure out which one is respon­si­ble for the decrease.

For exam­ple, sup­pose return fre­quency was flat, vis­its to impor­tant con­tent sky­rock­eted, but time on site fell through the floor.  You’ll prob­a­bly want to focus on time spent on site, and see if you can improve that.  But if your pri­mary KPI of leads/visits has increased (i.e. your con­ver­sion), maybe you’ve actu­ally done a really good thing and you should leave it alone. You’ve cre­ated a more fric­tion­less expe­ri­ence, and the declin­ing Vis­i­tor Engage­ment score sup­ports this.

At this point, you’ll face the unde­sir­able task of con­vinc­ing your execs that the Vis­i­tor Engage­ment met­ric, which you fought so hard to social­ize and adopt, should actu­ally decline.

But WAIT! Not all uber met­rics are bad
So I think you get the point.  Vis­i­tor engage­ment for­mu­las are largely another fad, just like para­chute pants and the Hol­ly­wood diet.  It’s a mea­sure some con­sul­tants and ven­dors can pitch like snake oil.

But, that is not to say that uber met­rics are com­pletely worth­less.  In select cases, you can actu­ally lever­age uber for­mu­las to make very use­ful decisions.

Uber met­rics that are purely objec­tive can hold value to an orga­ni­za­tion.  Per­haps one of the great­est is RFM – Recency Fre­quency Mon­e­tary.  In that case, you’re deal­ing with an (almost) entirely objec­tive uber met­ric.  For those not as famil­iar with RFM, it’s a clas­sic cus­tomer seg­men­ta­tion tech­nique that essen­tially calls for you to score your cus­tomers based on their ‘rel­a­tive’ rank to one another along three pri­mary metrics.

You then roll up these scores to arrive at an uber score, and iden­tify your best (high­est scor­ing) and worst (low­est scor­ing) cus­tomers.  Action you can take from learn­ings gleaned from this analy­sis are too numer­ous to name.  It’s actu­ally a lot of fun to do these kinds of models. But even in this case, sub­jec­tively can often enter the picture.

For exam­ple, the time­line over which you ana­lyze cus­tomer data is one of the prin­ci­pal points of sub­jec­tiv­ity.  In the RFM model above, do you ana­lyze behav­ior over 1 month, 6 months, 1 year or 6 years?   Maybe you just take as much data as you can find and mix it all together and hope for the best.  In turn, once you com­plete your RFM seg­men­ta­tion, what time period do you com­pare it to?  Weeks?  Months? Years?  Again, subjective.

RFM has a long his­tory of being valu­able – so again, I’m not throw­ing uber met­rics under the bus entirely.  Still, I wouldn’t waste your time with most of them.  There are so many oppor­tu­ni­ties for opti­miza­tion based on pri­mary key per­for­mance indi­ca­tors like con­ver­sion that you can keep your entire team busy for years.

Don’t try to build a bet­ter mouse trap, when you’re not tak­ing advan­tage of the one you’ve got today.

So, those are my thoughts.  As always, I wel­come your ideas and feedback.

13 comments
galerie-lounge
galerie-lounge

I think uber metric numbers make it easy to share quick information

Manthan
Manthan

RFM model has indeed come a long way to measure customer loyalty and build more targeted loyalty marketing programs. A customer who has purchased recently and frequently and created a high monetary value through these purchases is much more likely to purchase again. Such customers are called high RFM customers. On the other hand, customers who have not purchased in a long time tend to be comparatively less interested in the store/brand. Adding the counts for recency, frequency, and monetary value presents a good indicator of interest in the store/brand at the customer level. This is valuable information for a retail business to have. For more details on using RFM model, visit: http://thoughts.manthansystems.com/rfmmodel_business_analytics.php

Matthew Paxman
Matthew Paxman

Matt, It seems that the whole issue is not really a combination of metrics, in and of itself, but the built-in assumptions that bias the system. What we need to do is tell the system what means end-of-the-day conversion to us, and then let IT tell us how our customers are getting there (and where we have created dead-ends and roadblocks, killing conversion, or really turning on the green light to let the good times roll). After all, the whole point of analytics is to give yourself insight to make decisions, not make just make decisions and then look at pretty numbers on a page afterword. Also, the other problem encountered is how vague the outcome has become. Does Visitor Engagement, or your uber-metric d'jour, even mean your objective as an organization was reached? not only is the outcome subjective, but it is wrapped-up, multiplied, normalized, starched and ironed to mean zilch in and of itself. We need data values that speak to us, not buzz-worded cure-all wonder metrics. The take away? Don't tell the system what to give you back, rather let the system tell you what engaged and non-engaged Visitors and Customers are doing and use those values in your measurement. And above all, make sure to have a running analysis of what the analytics are telling you so you can adjust what values you have pulled into your formula and thus be flexible so far as the dynamic of your Visitors and Customers defines.

Cory Hendrickson
Cory Hendrickson

I have had the opportunity to work with both engagement metrics and RFM. While I agree that RFM is a valuable measure - the world of loyalty is solely based on this metric - it has not adapted to a digital world. The challenge here is in customer segmentation. There is little flexibility in an RFM model to account for external behavior. We assume that web properties remain islands competing (somewhat) equally for time. Today, this is not true. Conversion does not always happen at a single domain. Conversion is a product of both explicit search and implicit discovery. Where RFM is domain and customer dependent, engagement metrics aim to provide algebraic functions to analyze strengths and weaknesses against larger (and often unidentified) segments. Successful engagement metrics do not measure the engagement for one customer like defined here. A successful engagement metric is comprised of various inputs to help analysts identify strong channels. Perhaps some referring sites provide better conversion than others. Maybe time spent on a site is the greatest indicator of success. Engagement provides for loose interpretation of both domain dependent and domain independent customer events to help identify the proper metrics that are most important for any given business. While certainly not appropriate for an executive's daily dashboard report, they are the only current solution I have seen that answers the question - Why do we care about this specific metric more than the others?

David
David

I had an interesting discussion today with a client about a project to improve their conversion rates where they wanted immediate engagement. Two points about this one is that the client is having to calculate the ROI of the engagement on the value of the client now, as the product is subscription software it too difficult to calculate the life time value of the client as you cannot predict how long a client will continue to use your product. If the product is a yearly subscription it can provide feedback for the following year as to those who came from the campaign who continued with subscription. The problem is having a long enough view of your sales and how making decisions that will affect your potential sales for this year now without this data. Another client can use the visitor engagement around the number of returning visitors as a posistive behaviour and how this relates to who order an information pack. Can returning visitors be a unchanged visitor engagement metric that is consistent between industries?

Amadeus Fagereng
Amadeus Fagereng

I am all new to this, and it may be a stupid comment, but I will post it anyway. I do not see the problem with subjective parameters, as long as the formula is customized towards the specific company/website. After working with the company/website for a while you should know what the frequency "often" would be for your site. By using your previous analysis you will be able to produce an overall measurement of your sites performance without having to performing repetitive manual and subjective analysis. Also, the time it takes to come up with a measurement is crucial, and by having such an indicator you can deliver live performance. Of course, this is not the same as saying the formula would replace other methods, but I believe it would be a good indicator on when you want to look into other methods. The number would serve as and indicator of change in performance. I believe it is crucial to have such live (or close to) measurements that actually take into account subjective parameters/metrics of your website.

David Scoville
David Scoville

Thanks for the information. I think uber metric numbers make it easy to share quick information with executives or clients--to show them basically where they were then, and where they are now. I do agree with you on the subjectivity of uber metrics...and I'm still trying to swallow RFM.

Matt Belkin
Matt Belkin

James, Thanks for your comments. Did you happen to read the last paragraph of my blogs post? I actually talk about RFM and the value of that approach, which ties precisely into your thoughts. If have you haven't done so, please read that part of my post and let me know if you have questions. Thx, Matt

Dr. James Joseph Geertz
Dr. James Joseph Geertz

Matt, I would be curious as to who the company was that you found producing "visitor engagement", as well as which metrics, if any, they were employing subjectively and which metrics, if any, they were employing objectively. Your posting doesn't make it clear that you actually performed any serious analysis of the formula before dismissing it as an inaccurate uber formula. Rather you jumped from calling the formula a mashup of metrics to berating a hypothetical formula of metrics mashups. There is a wonderful world of mathmatics out there. It is short-sighted to believe that, with all the data that can be collected by site hosts, the end all be all of web analytic tools is going to be a division problem with two counting statistics (leads/visits). If you are not going to throw the formula down on the mat, display it in the post, and point out the flaws, it is hard to understand how your view is not simply a heavy-handed dismissal of all non-traditional statistics. Hopefully Omniture is not nearly so close minded. Regards, James

CMS Specialist
CMS Specialist

Do you have any experiences on creating realtime feedback on the scores you are calculating?

Nathan Janitz
Nathan Janitz

While in principle I disagree with Matt, I see where he is coming from with the “engagement” metrics being very subjective. But as Steve pointed out, “As a consultant using your tools it’s our responsibility to make or save our clients money. I often find that in order to do that I have to do a lot more than simply measure conversion funnels because all that does is show low hanging fruit which will only give you so much (and only works for some sites).” One can test until they are blue in the face, but at some point we have to answer the questions: “why is this not working”. You cannot do this without measuring some form of engagement. Leaders in this industry don’t become leaders by just giving up after the last test; they figure out a way to answer questions. They try to understand the why. While I agree with Matt’s statement about building a better mouse trap, you also can’t catch a mouse without looking at the right information….all of the right information. While 1 metric can determine absolute success (how does this make me money), it can never answer the questions “why”, “how”, or “so what.” As Steve said, looking at the problem through on KPI is worthless. I would also add looking at them without the right context is also worthless. Is 5 seconds or 5 minutes the right time-on-page? The answer completely depends on the context. 5 minutes on a checkout page or spending 5 seconds watching a 6 minute video can both give insight into how a person engages with the content/website. Someday the discussion of what defines “engagement” will become a thing of the past and be replaced with the next new idea. The methods themselves will stay and evolve as we test, evaluate, and test again (basically how web analytics itself has evolved). Again, at some point logic must take part and realize that by nature “engagement” is somewhat subjective, but can be qualified when put into the right context. If web analytics were as simple as just conversions, most web analytics platforms would be a single page with a pass/fail reading. The world is not always as black or white as we might like. It is our responsibility to make sure that we maximize our client’s money. As thought leaders in the industry, it is also our responsibility to make sure that we are looking at new ways of explaining what is happening on the website. The most recent way of doing that is by trying to define and measure engagement. It’s not a fad; it is an evolution of the industry... the fad is more likely the way we view/measure engagement.

Steve Jackson
Steve Jackson

Hi Matt, I had to chime in here because I am consistently seeing this debate appear again. This is a recent excerpt I posted on WebAnalyticsDemystifieds' Future of blog talking about the same thing; "It goes back to a point I first read about ages ago on Eric’s blog (late 2006) and then when Eric published his engagement formula became the legendary “engagement” debate I’m sure we all remember, on Occam’s Razor, Jims site, my site and a bunch of others. It got quite heated at times as it should. Passions were ignited and people were drawing lines in the sand. At the time I took a step back and looked at what we all were saying and came to the conclusion we were largely debating semantics though we all agreed on some things. What we pretty much all agree on is that the more ways we have of identifying ways to get more customers and take actions on our metrics the better. Note the “take action” part. I think most of us agree if you can’t act on the KPI then why use it. We all agree that we have to use quantitative (clickstream), qualitative (voice of customer, attitudinal) and competitive (comparison) data to drive the best insights. Learning to combine these data sources is the way the industry will move. " I think you would agree if you can take action on a metric then you can use it judging by the points you make above. In the "legendary debate" Avinash argued the same point as you're making and I'm not disagreeing with either perspective here. The whole point of everything is conversion eventually. The problem then was one guy was discussing engagement in the context of RF models. One was discussing engagement in terms of bounce rate, one was discussing engagement in terms of scoring actions and I was defining engagement segments and everyone else was throwing their 2 cents into the mix till you had a whole big bag of ideas labeled as engagement. I have used Omnitures (and other) tools successfully to save clients millions of euros using nothing but engagement segments as I define them. No conversion was made initially but the likelihood to convert later (measured via RF) was much higher and by using engagement indexes to optimize keyword spend and banner placements we saved clients a lot of money. As a consultant using your tools it's our responsibility to make or save our clients money. I often find that in order to do that I have to do a lot more than simply measure conversion funnels because all that does is show low hanging fruit which will only give you so much (and only works for some sites). The puzzle I am being asked to solve more and more is this. How come multi-channel campaigns for the past 12 months have shown only average or unremarkable conversion results and yet our profits are still rising exceptionally? Which part of our ad money are we wasting? Something is working but which is it? In the above situation conversion rate alone is worthless. Much like engagement index alone, or bounce rate alone or page views alone. The point I think folks might be missing is that it's not about "one metric to measure all" it's about taking various metrics in context to each other and knowing how to interpret and act on those metrics. Best, Steve.

Jim Novo
Jim Novo

Hi Matt - I'm with you on this in terms of the practicality of explaining and acting on these complex models. While the person who contructs the model may be able to "see it in their head", it's tough to press forward from there in the org. And I would also argue that RFM was the very first engagement model and still a wildly successful one at that, though for the purposes of the web we might drop the "M" so we get a model that applies outside of commerce. With RF, now you're down to two questions: How many times did they act (F) and how long ago was the last action (R)? Simple to explain to just about anyone and the R component is strongly predictive of likelihood to act in the Future, which gets us into predicting the Potential Value of a Visitor / Customer. In the long run, I think people will find that measuring dis-Engagement (Recency) is much more important than measuring Engagement, because it tells you the highest ROI point to act. People who are Engaged don't need much Marketing. People who have dis-Engaged are a lost cause. People who are in the process of dis-Engaging, if you can catch them, is where you can make a lot of money.