Last week we looked at dif­fer­ent ways to orga­nize your report suites to pro­vide the proper level of infor­ma­tion to your dif­fer­ent stake­hold­ers.  This week I’d like to take a deeper look at the inside of sin­gle report suite and how its lay­out fits into the over­all scheme.

Per­son­al­iz­ing Your Data

All of the reports in Site­Cat­a­lyst cor­re­spond to a data­base table some­where.  Some of these are pop­u­lated out-of-the-box and some are left to the dis­cre­tion of a com­pany to decide what to use them for and how to col­lect the data.

Among these per­son­al­ized vari­ables are props, evars (also here), and events.  That pro­vides a lot of room for cus­tomiz­ing the data you col­lect and mak­ing sure that you’re meet­ing all the goals laid out in your mea­sure­ment strat­egy.

When you orga­nize your report suites into a hier­ar­chy, you have to keep in mind that a sin­gle data point is likely going to end up in mul­ti­ple loca­tions.  A Video Start on the US site will also end up in the global site.  It sounds pretty straight­for­ward, but there are some impor­tant effects that we’ll take advan­tage of.

1.  Some Of Your Data Is Only Impor­tant At The Top Level

There are some reports that are only impor­tant inside the upper lev­els of your hier­ar­chy.  Inside the Omni­ture Suite for exam­ple, we cap­ture Prod­uct in prop4/evar4.  Every time data is sent to the col­lec­tion servers, prop4 and evar4 con­tain a value that sim­ply iden­ti­fies which prod­uct is being used: Site­Cat­a­lyst, Test&Target, Dis­cover, etc.

Within an indi­vid­ual report suite, the value is lim­ited, but at the global level, where all the data is, this is extremely valu­able.  It pro­vides me a sin­gle report that tells me how much usage each prod­uct is get­ting.  Inter­est­ing insights are avail­able from watch­ing this trend over time.

This con­cept will hold true regard­less of whether you’ve orga­nized your hier­ar­chy by coun­try, by prod­uct, by site sec­tion, or some­thing else.  Some of your data is only impor­tant when you look from the top down.

2.  Some Of Your Data Is Only Impor­tant At The Bot­tom Level

We keep track of the usage of bid rules used for ad spend opti­miza­tion, but Search­Cen­ter is the only prod­uct that uses them.  An active cam­paign in Test&Target is its own event and isn’t repeated any­where else through­out the suite.  We obvi­ously want to keep track of these fea­tures, how peo­ple use them, and how effec­tive they are.

We keep track of these kinds of things at the bot­tom level of the hier­ar­chy, but we don’t need to expend the effort to mea­sure them at the top level.  There is no added value in look­ing at this data at the top level, there­fore, there is no need to track it at the top level. In fact there is no need to even turn these vari­ables on at the top level.

Now here’s the neat trick.  Since we don’t roll these val­ues up to the global level, it means that the lower lev­els can re-use the same vari­ables to mean dif­fer­ent things with­out cor­rupt­ing any of your data.

3. Some Of Your Data Is Just Plain Important

This fol­lows log­i­cally from points 1 and 2, but I thought I’d call it out just to be explicit.  Some of your data makes sense and pro­vides value at the top, at the bot­tom, and every­where in between.

Take­aways

When you deter­mine that you need to mea­sure a par­tic­u­lar data point, you must decide where it has value, and at which level you need to keep track of it.

If a data point falls into cat­e­gory 1 (pro­vides value at upper lev­els only), then you must ensure that every report suite under­neath imple­ments that data point con­sis­tently.  This means they must pass the same val­ues (or kinds of val­ues) as every other report suite, and they must remain con­stant over time.

If a data point falls into cat­e­gory 2 (pro­vides value at lower lev­els only) then record it in a way that is con­sis­tent within that report suite, and do not even enable that vari­able at the upper lev­els.  This reduces risk of latency and also ensures that your users do not waste time try­ing to derive value from reports that have none.

Inside the Omni­ture Suite

As an (extremely sim­plis­tic) exam­ple, the dia­gram below shows a lay­out of our vari­ables inside a report suite.  If it’s listed on a report suite, it’s being recorded there.  Orange text is unique to a given report suite and not recorded at a global level.  Faded text are the ones you would pay less atten­tion to on a given level.

Cat­e­gory 1 — Inside the Omni­ture Suite, we have ded­i­cated about half of our vari­ables (props, evars, and events) that must be used con­sis­tently across all prod­ucts.  There are global stan­dards for imple­ment­ing these vari­ables so that the val­ues mean the same thing and that they look the same when they come in.  Prod­uct Name is an example.

Not all 35 data points apply to every prod­uct, but they do apply to more than one.  If a par­tic­u­lar data point does not apply to a prod­uct, we sim­ply leave it alone.  Report Suite ID is one exam­ple.  We obvi­ously want to record that for Site­Cat­a­lyst, Gen­e­sis, Dis­cover, etc., but it just doesn’t apply to Test&Target or Insight. File For­mat is another exam­ple.  We want to know what for­mats are most pop­u­lar when peo­ple down­load and sched­ule reports, but you can’t do that in Test&Target.  So Test&Target just doesn’t set that variable.

Cat­e­gory 2 — The vari­ables that are not used in cat­e­gory 1 get used in cat­e­gory 2.  These are cus­tomized to the needs on an indi­vid­ual prod­uct and are not enabled at the upper level.  Search­Cen­ter bid rules, Dis­cover work­space seg­ments, and Test&Target active cam­paigns are good exam­ples.  Again, note that within this cat­e­gory, I can reuse the same vari­able to mean dif­fer­ent things since they don’t roll up to the global level.

Cat­e­gory 3 — You can actu­ally treat these as a sub­set of cat­e­gory 1 because the same rules apply.  They must be imple­mented con­sis­tently through space and time for all prod­ucts.  Login Com­pany is an exam­ple that pro­vides value at the indi­vid­ual level and at the aggre­gate level.

It was Ben­jamin Franklin that said “An ounce of pre­ven­tion is worth a pound of cure.”  A week or two ago I men­tioned the measure-once-cut-twice phi­los­o­phy that could be restated by a web ana­lyst to say that “an ounce of plan­ning is worth a pound of devel­op­ment.”  Or maybe it’s worth a pound of headache.  Maybe it’s worth more.

2 comments
Ben Robison
Ben Robison

Yeah, this is one of those things are that really hard to keep track of. I think part of the reason is that it's actually fairly straightforward to say and to visualize. There's no problem seeing the big picture, but the challenge is in figuring out how to keep track of dozens (hundreds) of individual report suites. I've got a spreadsheet to keep track of what we're tracking where, that seems to work pretty well as long as I remember to keep it updated.

Jim Humphrys
Jim Humphrys

Nice post, Ben. This is an area where I've struggled to maintain control and your visualization will help.