Latency in SiteCatalyst 15

One of the most talked-about changes from Site­Cat­a­lyst 14 to Site­Cat­a­lyst 15 is the way data is processed and how long it takes until activ­i­ty on the site is vis­i­ble in the report­ing — the so called “laten­cy”.

Site­Cat­a­lyst col­lects data from web sites or apps. It then process­es the data and cre­ates a mul­ti­tude of reports, all read­i­ly avail­able in the Site­Cat­a­lyst user inter­face. Data flows in and gets aggre­gat­ed so it makes sense.

Our engi­neer­ing teams have made sig­nif­i­cant changes in Site­Cat­a­lyst 15, mean­ing Site­Cat­a­lyst 14 and Site­cat­a­lyst 15 process data in an entire­ly dif­fer­ent way.

Let’s look at two myths that I hear a lot:

SiteCatalyst 14 is real-time”

That is part of the truth, but not the com­plete pic­ture.

Data pro­cess­ing in Site­Cat­a­lyst fol­lows the prin­ci­ple “process as soon as you can”.

When a vis­i­tor calls up a web page, her brows­er sends a track­ing request. Site­Cat­a­lyst will almost imme­di­ate­ly count this as a Page View. The vis­it, how­ev­er, is count­ed once it’s fin­ished, mean­ing after 30 min­utes of inac­tiv­i­ty.

As a result, some of the reports in Site­Cat­a­lyst are prac­ti­cal­ly real-time, the most pop­u­lar exam­ple being the Site Con­tent > Pages report. Cus­tom Traf­fic reports also dis­play real-time data for the Page Views met­ric.

Oth­er reports are at least 30 min­utes behind, because the sys­tem will not process vis­it-based met­rics before the vis­it has actu­al­ly timed out. The same goes for some of the con­ver­sion reports and met­rics.

SiteCatalyst 15 has a latency of 2 hours”

That is not accu­rate.

Site­Cat­a­lyst 15 process­es data in batch­es. Those batch­es are usu­al­ly 60 min­utes of col­lect­ed data. As an exam­ple, all track­ing data that arrives into Site­Cat­a­lyst between 11am and 11:59:59am is part of the same batch.

At 12:00, Site­Cat­a­lyst starts col­lect­ing data into a new batch. At the same time, the “11am batch” goes into pro­cess­ing. Pro­cess­ing of a 1-hour batch cur­rent­ly takes around 30 min­utes.

So, when do we see data?

If a track­ing request came in at 11:00:01am, it’ll be part of the “11am batch” which will fin­ish pro­cess­ing at around 12.30. The data will there­fore be vis­i­ble just under 90 min­utes after it hap­pened.

If, on the oth­er hand, the track­ing request came in at 11:59:59am, it’ll also be part of the “11am batch” and there­fore also vis­i­ble at 12.30, a mere 30 min­utes after it hap­pened.

As a result of the batch approach, laten­cy looks like this:

[Screenshot]

Laten­cy in Site­Cat­a­lyst 15 — 60 ver­sus 30 minute slices

Notes

Not all data is processed in 60 minute batch­es. The batch size can be changed to 30 min­utes for crit­i­cal report suites (2 per cus­tomer).

Because a 30 minute batch can be processed in rough­ly 15 min­utes, the max­i­mum time for a hit to appear in Site­Cat­a­lyst is 45 min­utes, and the short­est time is 15 min­utes, fol­low­ing the same saw-tooth pat­tern as above.

What if you are fol­low­ing web activ­i­ty for a new release or after hav­ing sent a newslet­ter? What if your edi­to­r­i­al team needs to know right now how that new arti­cle about that politi­cian per­forms?

Site­Cat­a­lyst 15 now has “Cur­rent Data” reports which work a lot like Site­Cat­a­lyst 14 worked: each report in Site­Cat­a­lyst 15 has a “Cur­rent Data” coun­ter­part which pro­vides the same laten­cy that Site­cat­a­lyst 14 did.

[Screenshot]

Cur­rent Data Reports in Site­Cat­a­lyst 15

Those reports can be used to check data fo the cur­rent day or cur­rent day and the day before.

If you need Cur­rent Data reports, go to Admin > User Man­age­ment and add a user to the “Cur­rent Data” group. That user will now be able to see those reports.

[Screenshot]

Cur­rent Data Reports Group

3 Responses to Latency in SiteCatalyst 15

  1. barry mann says:

    Hi

    Excel­lent arti­cle, par­tic­u­lar­ly for those in the pub­lish­ing sec­tor for whom 2 hours just won’t cut the mus­tard.

    Next ques­tion: We have been up-dimen­sion­ing many evars from 500k to 1m or more. I was told that this “intro­duces fur­ther laten­cy”. If we wind it up fur­ther to 2m or even 5m laten­cy will increase fur­ther (I am told). Why?

    • Jan Exner says:

      Hi Bar­ry,

      Glad this makes sense!

      To your ques­tion: upping uniques does not affect the laten­cy this arti­cle is about. It does affect report­ing speed. To illus­trate this: if you up your lim­its too high, reports in Site­Cat­a­lyst might not come back before the UI times out, and you’ll essen­tial­ly have to pull the lim­its back down a bit to be able to use that report at all.

      In my expe­ri­ence, 2m is push­ing your luck, but it does work for some cus­tomers.

  2. Pingback: Debugging | Web Analytics for Developers

Leave a Reply

Your email address will not be published. Required fields are marked *