Top Ten Tips to Minimize Data Latency [Inside Omniture SiteCatalyst]

Analytics · By On · 11 Comments

One of the many rea­sons cus­tomers choose Omni­ture is our real-time report­ing capa­bil­i­ties to pro­vide that rapid insight needed to make deci­sions in a fast-paced online busi­ness envi­ron­ment.   Our engi­neer­ing teams work hard to ensure we con­sis­tently deliver this capa­bil­ity, but there are rare occa­sions where latency has pre­sented chal­lenges for some cus­tomers.  It’s impor­tant to note with any latency issue that no cus­tomer data is lost and com­plete data integrity is main­tained.  While we’ve found that recent latency issues have only affected a minor­ity of cus­tomers, it brought to light some other poten­tial con­tribut­ing fac­tors that you can con­trol such as using too many server-intensive fea­tures within a report suite.  These are things that you can do some­thing about, so in this post I will explain steps you can take to help min­i­mize data latency.

A Lit­tle His­tory…
As I men­tioned, latency can be harder to recover from if you’ve enabled cer­tain Site­Cat­a­lyst fea­tures.  Some Site­Cat­a­lyst fea­tures do not have much of an impact and oth­ers do.  Unfor­tu­nately, many clients do not know which fea­tures are resource inten­sive and which are not.  In the old, old days (when I used Site­Cat­a­lyst ver­sion 9), if you wanted to turn on a fea­ture, such as a 2-item Traf­fic Data Cor­re­la­tion, you would need to con­tact Omni­ture and pay for it in an ad-hoc man­ner.  In fact, almost every­thing you wanted to do meant a phone call, an adden­dum and an account man­ager.  Cus­tomers com­plained that it took too much time and that Omni­ture was “nickel and dim­ing” its cus­tomers so Omni­ture moved to the Admin­is­tra­tion Con­sole model where cus­tomers were empow­ered to turn many things on and off as desired.  In fact, many cus­tomers ask if ALL func­tions can be added to the Admin­is­tra­tion Con­sole, but doing this could allow cus­tomers to enable things that could pro­duce latency.

Finally, before dig­ging into the details, let me be clear that it is our inten­tion for Omni­ture cus­tomers to use all of the fea­tures Site­Cat­a­lyst pro­vides.  While the items below may cause you to think about using fea­tures I have pre­vi­ously raved about on this blog, keep in mind that I am pro­vid­ing this infor­ma­tion only so that you under­stand what taxes the sys­tem the most.  If you have a good busi­ness case for using any of these fea­tures, please do so.  We often find that Omni­ture cus­tomers have enabled fea­tures that are both resource-intensive and not actu­ally being used, caus­ing a lose-lose sit­u­a­tion. When I was a cus­tomer, I would review my imple­men­ta­tion a few times a year and find out if any of my users were actively using the fea­tures that were enabled and take action accordingly…

#1 — Full Sub­re­la­tions
While Con­ver­sion Vari­able Sub­re­la­tions are extremely pow­er­ful, it is impor­tant that you plan ahead and think about what eVars for which you need full sub­re­la­tions since they cre­ate an enor­mous amount of over­head.  If you think about it, if you have 10 eVars enabled and then turn on full sub­re­la­tions for one eVar, you are instruct­ing Site­Cat­a­lyst to cre­ate nine new data cubes, all of which have to be updated every time a Suc­cess Event takes place.  For those of you that have access to Omni­ture Dis­cover, I would encour­age you to lever­age this tool for advanced vari­able break­downs instead since it is built for this capa­bil­ity.  Some other solu­tions to this are to con­cate­nate val­ues to one eVar and use Clas­si­fi­ca­tions to split them out or to use a Traf­fic Data Cor­re­la­tion which is much less resource intensive.

#2 — Unused Con­ver­sion Vari­ables
Did you know that even if you are not pass­ing any data to a Con­ver­sion Vari­able (eVar) it is increas­ing your pro­cess­ing load?  This is due to the fact that every time a Suc­cess Event takes place, it is asso­ci­ated to every enabled eVar, even if the value asso­ci­ated with it is “None.”  There­fore, when I see clients that have 20 eVars enabled, but are only actively using 2–3 of them I beg them to dis­able the eVars that are not being used so they can improve performance.

#3 — Mul­ti­ple Suc­cess Events on a Page
Another item that can impact latency is the set­ting of mul­ti­ple Suc­cess Events on a page.  If you set two Suc­cess Events on a page, you are essen­tially dou­bling all of the data pro­cess­ing men­tioned above on those pages.

#4 — Traf­fic Spikes
While it is not pos­si­ble to antic­i­pate every traf­fic spike your site will have, there are many cases in which you know that you will receive more traf­fic such as hol­i­day peri­ods or via sea­son­al­ity.  Site­Cat­a­lyst pro­vides a way for you to tell Omni­ture when you expect a traf­fic spike so we can be pre­pared for it by adding addi­tional resources.  This noti­fi­ca­tion sys­tem can be found in the Admin­is­tra­tion Console:

#5 — Par­tic­i­pa­tion
A few weeks ago, I explained Par­tic­i­pa­tion in a post.  While Par­tic­i­pa­tion is very pow­er­ful, you prob­a­bly do not need to have Par­tic­i­pa­tion enabled for every Suc­cess Event.  For exam­ple, if you have a five step con­ver­sion fun­nel and set a Suc­cess Event for each step, you could prob­a­bly be ok with enabling Par­tic­i­pa­tion on the first and last step with­out los­ing much insight.  Also, once you use Par­tic­i­pa­tion to learn about what pages lead to your site Suc­cess Events, it may be ok to dis­able Par­tic­i­pa­tion and re-enable it later in the year or after a site/page redesign since the data may not change dras­ti­cally from month to month.

#6 — Vis­its & Vis­i­tors in Con­ver­sion Reports
A fre­quent request from cus­tomers is to allow Vis­its and Unique Vis­i­tors in Con­ver­sion reports so they can be used in Cal­cu­lated met­rics.  I often solve this prob­lem by pulling data into the Excel­Client where I can com­pare Traf­fic Vis­its for a day to Con­ver­sion Suc­cess Events for the same date.  How­ever, if this func­tion­al­ity is crit­i­cal to your busi­ness I sug­gest you talk to your Account Man­ager or Omni­ture Consulting.

#7 — Com­plex VISTA Rules
When writ­ing VISTA rules, some­times there is a ten­dency to throw every­thing in (includ­ing the kitchen sink!).  The more com­plex you make VISTA rules, the slower data may process.

#8 — Unique Val­ues
There are some Traf­fic Vari­ables (sProps) and Con­ver­sion Vari­ables (eVars) that end up hav­ing very few unique val­ues and oth­ers that have mil­lions of unique val­ues.  For exam­ple, an eVar cap­tur­ing the user’s birth year will have a rel­a­tively finite num­ber of val­ues, whereas an eVar cap­tur­ing unique User ID’s for a site like Face­book, would have mil­lions of unique val­ues.  While this does not impact pro­cess­ing speed, it can impact report­ing speed.  For this rea­son, you need to think about how many unique val­ues you expect a vari­able to have when set­ting up Con­ver­sion Vari­able Sub­re­la­tions and Traf­fic Data Cor­re­la­tions.  Many times, cus­tomers will use a VISTA rule to “blank out” vari­ables in the Site­Cat­a­lyst inter­face that have over 500,000 unique val­ues per month and lever­age the DataWare­house for these data points.

#9 — Prod­ucts Vari­able
In a past post I men­tioned dif­fer­ent ways that you could use the Prod­ucts Vari­able in Site­Cat­a­lyst.  Since the Prod­ucts Vari­able is cur­rently one of the only Con­ver­sion Vari­ables that allows you to pass in mul­ti­ple val­ues, I have seen some clients be very cre­ative with its use.  How­ever, pass­ing in more than ten items to the Prod­ucts vari­able can cause prob­lems so if you have a need for stor­ing many items in the prod­ucts vari­able, I sug­gest you speak to your Account Man­ager or Omni­ture Con­sult­ing.

#10 — Event Seri­al­iza­tion
In a recent post I explained how you can use Event Seri­al­iza­tion to de-duplicate Suc­cess Events.  How­ever, if you have mil­lions of Suc­cess Events tak­ing place on your site, using Event Seri­al­iza­tion can become a resource issue.  It is best used for Suc­cess Events that are crit­i­cal to your site, not every Suc­cess Event.

So there is your top 10 list of cul­prits.  Omni­ture is con­stantly look­ing at what it can do to improve per­for­mance and hopes that many of the items above will be less resource inten­sive in the future, please keep these items in mind when talk­ing to your Account Man­ager or ClientCare.

Have a ques­tion about any­thing related to Omni­ture Site­Cat­a­lyst?  Is there some­thing on your web­site that you would like to report on, but don’t know how?  Do you have any tips or best prac­tices you want to share?  If so, please leave a com­ment here or send me an e-mail at insidesitecatalyst@​omniture.​com and I will do my best to answer it right here on the blog so every­one can learn! (Don’t worry — I won’t use your name or com­pany name!).  If you are on Twit­ter, you can fol­low me at http://​twit​ter​.com/​O​m​n​i​_​man.

Learn more about Omni­ture Consulting
Learn more about Omni­ture University
Tagged with → • •  
  • Dan Roden

    I get what you are say­ing about pro­cess­ing, but from a customer’s view, hav­ing been “sold” all of these rich fea­tures as solu­tions to our RFP’s and other WA require­ments, you are basi­cally say­ing that we should expect latency if we use any­thing that is required for high level ana­lyt­ics mea­sures to answer enter­prise level questions.

  • Sal

    I think that the main issue with the latency is not as much the latency, but that instead of report­ing latency issues, Omni­ture reports just return incor­rect val­ues. We have reg­u­larly had to go back and change reports and dash­boards, as we only later learned that there were server latency issues. With the latency being unpre­dictable, is there any good solu­tion for us to avoid uti­liz­ing bad data?

  • http://blogs.omniture.com/author/agreco Adam Greco

    Dan — I know this is a del­i­cate issue, but if you look at my dis­claimer, the only pur­pose of this post is to make peo­ple aware of what taxes the sys­tem the most, not to dis­suade you from using them if you have a valid busi­ness objec­tive. Many cus­tomers enable things that they don’t actively use and then won­der why things are slow..I hope that makes sense…

    Adam

  • http://blogs.omniture.com/author/agreco Adam Greco

    Sal — Any sit­u­a­tions like you are describ­ing below should be reported to Client­Care immediately…Thanks!

    Adam

  • http://www.ewm.co.uk Sandy Miller

    Hi I do not regard your arti­cle as help­ful. Our Omni­ture report­ing has been suf­fer­ing greatly from severe latency — to the point taht I am con­sid­e­ing employ­ing sec­ondary ana­lyt­ics since Omni­ture gives nowhere near real-time data.

    ‘Nor­mal’ ser­vice is pretty close to real-time but we do not seem to receive ‘nor­mal’ ser­vice very often or for any sus­tained period of time. I cant check my sales for exam­ple, or any cam­paigns data sim­ply because Omni­ture is so slow. This is, I would sug­gest, the sec­ond major out­age in almost as many months — we suf­fered greatly in Jan­u­ary. And now, for the last week, data is con­stantly delayed roughly 8 hours. Use­less for proac­tive man­age­ment of my ecom­merce enterprise.

    Not that ‘nor­mal’ was ever that good either. Sorry, but we have had noth­ing but prob­lems with Omni­ture from the start. The fact I cant see any of todays data ren­ders your sys­tem redundant.

    Any com­ments Omni­ture?? You can email me. Oh, and I have been com­mu­ni­cat­ing these issues con­stantly to our part­ner
    http://​www​.​e​-inbusi​ness​.co​.uk

    sandy.​miller@​ewm.​co.​uk

  • http://blogs.omniture.com/author/agreco Adam Greco

    Sandy,

    First, I apol­o­gize that you are hav­ing latency prob­lems. I have asked Omni­ture­Care to follow-up with you imme­di­ately to see if there is any­thing else that can be done. Hav­ing been a client myself, I under­stand how frus­trat­ing latency can be and I assure you that we are doing every­thing in our power to address the issue. That being said, the pur­pose of this post was sim­ply to let clients know about fea­tures that are more server-intensive than oth­ers to be sure that clients double-check that they are actively using them. We find many clients have our most server-intensive fea­tures enabled, but are not using them. The feed­back we received at Omni­ture Sum­mit was that cus­tomers did not know which Site­Cat­a­lyst fea­tures took up more server resources than oth­ers and this post was aimed at com­mu­ni­cat­ing this so Omni­ture can be as trans­par­ent as pos­si­ble. Thanks!

    Adam

  • http://www.ird.govt.nz Wei

    Hi can any­body tell me whether Omni­ture cus­tomers can host this web ana­lyt­ics tool in-house? Are any clients doing so? What oper­at­ing plat­form can it be run on and what data­base is need? Thanks.

  • http://blogs.omniture.com/author/agreco Adam Greco

    Wei — You can run Omniture’s Dis­coverOn­Premise in-house, but no SiteCatalyst.

    Adam

  • http://learn-n-optimize.blogspot.com/ Sushant Ajmani

    Here are cou­ple of more con­tribut­ing fac­tors towards Latency: -

    SAINT Clas­si­fi­ca­tion — Though this is among the best fea­tures of Site­Cat­a­lyst but, it can seri­ously impact the reports per­for­mance if you go beyond 10 Clas­si­fi­ca­tion Attrib­utes. Last year, we setup 25 Clas­si­fi­ca­tion Attrib­utes on the request of our client who was very pushy, and it causes train wreck per­for­mance. Some of the reports took more than 40 mins to load. So; if you are using clas­si­fi­ca­tion to clas­sify your Prod­ucts, Cam­paigns, Con­tent and Store IDs, please keep the # of attrib­utes under 10 oth­er­wise, you will suf­fer sever per­for­mance issues and often times, your site will show latency issues.

    Cus­tom Link Track­ing — This is often used with AJAX Com­po­nents where, the engage­ment is more impor­tant than the actual page view activ­ity. Most of the retail­ers these days, spend a decent amount of money cre­at­ing these fancy wid­gets and wants to track each and every click and mouseover activ­ity, and if you are trig­ger­ing Cus­tom Link Track­ing code at a reg­u­lar inter­val, you are not only impact­ing the per­for­mance of the wid­get, but might also result in latency. I don’t have good data to prove but, look­ing for­ward for that.

    CMS Con­tent on the site — Most of the Brands and Retail­ers have lots of con­tent on their Home, Cat­e­gory and Shop Pages which takes a lot of time to load and if your Omni­ture code is at the extreme footer then, it might impact your Image Request response time. Recently, I observed 74sec response time which was heart break­ing as com­pare to a bench­mark num­ber of 100-180msec

    Packet Header Size — There are lot of retail­ers these days uses more than 1 ana­lyt­ics solu­tion on their site, and the best com­bi­na­tion you would find on the inter­net is Omni­ture and Google Ana­lyt­ics. Both allows the First Party Cookie setup as per the indus­try best prac­tices but often times, the tech­nol­ogy folks doesn’t keep an eye on the packet header which con­tains a lot of infor­ma­tion which goes beyond the thresh­old of Omni­ture and they split up the pack­age in to 2 which causes another set of per­for­mance issues and slower response time.

    Prob­lem caused by an Unknown Client on the Server — If your site is on a same server where 30 other clients of Omni­ture resides, and one of them is fac­ing a seri­ous latency issue then, you might also become a vic­tim of that.

  • http://LATIMES.com Rob Pat­ton

    It would be a great improve­ment if Site­Cat­a­lyst could dis­play a latency warn­ing that is promintently vis­i­ble when­ever the servers arent caught-up to nom­i­nal near-raltime refreshes. This would pre­vent users from accidet­nally report­ing incom­plete data to their col­leagues. Sim­i­lar alerts would be help­ful if daily/weekly data extracts and other time-sensitive processes become delayed.

  • http://blogs.omniture.com/author/agreco Adam Greco

    Rob,

    I have passed your sug­ges­tion to Prod­uct Man­age­ment. Thanks!

    Adam