Blog Post:One of the most talked-about changes from SiteCatalyst 14 to SiteCatalyst 15 is the way data is processed and how long it takes until activity on the site is visible in the reporting - the so called "latency". SiteCatalyst collects data from web sites or apps. It then processes the data and creates a multitude of reports, all readily available in the SiteCatalyst user interface. Data flows in and gets aggregated so it makes sense. Our engineering teams have made significant changes in SiteCatalyst 15, meaning SiteCatalyst 14 and Sitecatalyst 15 process data in an entirely different way. Let's look at two myths that I hear a lot:

"SiteCatalyst 14 is real-time"

That is part of the truth, but not the complete picture. Data processing in SiteCatalyst follows the principle "process as soon as you can". When a visitor calls up a web page, her browser sends a tracking request. SiteCatalyst will almost immediately count this as a Page View. The visit, however, is counted once it's finished, meaning after 30 minutes of inactivity. As a result, some of the reports in SiteCatalyst are practically real-time, the most popular example being the Site Content > Pages report. Custom Traffic reports also display real-time data for the Page Views metric. Other reports are at least 30 minutes behind, because the system will not process visit-based metrics before the visit has actually timed out. The same goes for some of the conversion reports and metrics.

"SiteCatalyst 15 has a latency of 2 hours"

That is not accurate. SiteCatalyst 15 processes data in batches. Those batches are usually 60 minutes of collected data. As an example, all tracking data that arrives into SiteCatalyst between 11am and 11:59:59am is part of the same batch. At 12:00, SiteCatalyst starts collecting data into a new batch. At the same time, the "11am batch" goes into processing. Processing of a 1-hour batch currently takes around 30 minutes. So, when do we see data? If a tracking request came in at 11:00:01am, it'll be part of the "11am batch" which will finish processing at around 12.30. The data will therefore be visible just under 90 minutes after it happened. If, on the other hand, the tracking request came in at 11:59:59am, it'll also be part of the "11am batch" and therefore also visible at 12.30, a mere 30 minutes after it happened. As a result of the batch approach, latency looks like this: [caption id="attachment_221" align="aligncenter" width="300"][Screenshot] Latency in SiteCatalyst 15 - 60 versus 30 minute slices[/caption]

Notes

Not all data is processed in 60 minute batches. The batch size can be changed to 30 minutes for critical report suites (2 per customer). Because a 30 minute batch can be processed in roughly 15 minutes, the maximum time for a hit to appear in SiteCatalyst is 45 minutes, and the shortest time is 15 minutes, following the same saw-tooth pattern as above. What if you are following web activity for a new release or after having sent a newsletter? What if your editorial team needs to know right now how that new article about that politician performs? SiteCatalyst 15 now has "Current Data" reports which work a lot like SiteCatalyst 14 worked: each report in SiteCatalyst 15 has a "Current Data" counterpart which provides the same latency that Sitecatalyst 14 did. [caption id="attachment_227" align="aligncenter" width="250"][Screenshot] Current Data Reports in SiteCatalyst 15[/caption]Those reports can be used to check data fo the current day or current day and the day before. If you need Current Data reports, go to Admin > User Management and add a user to the "Current Data" group. That user will now be able to see those reports. [caption id="attachment_228" align="aligncenter" width="300"][Screenshot] Current Data Reports Group[/caption]
Author: Date Created:19 March 2013 Date Published: Headline:Latency in SiteCatalyst 15 Social Counts: Keywords: Publisher:Adobe Image:http://blogs.adobe.com/digitaleurope/files/2013/03/81.png

One of the most talked-about changes from Site­Cat­a­lyst 14 to Site­Cat­a­lyst 15 is the way data is processed and how long it takes until activ­ity on the site is vis­i­ble in the report­ing — the so called “latency”.

Site­Cat­a­lyst col­lects data from web sites or apps. It then processes the data and cre­ates a mul­ti­tude of reports, all read­ily avail­able in the Site­Cat­a­lyst user inter­face. Data flows in and gets aggre­gated so it makes sense.

Our engi­neer­ing teams have made sig­nif­i­cant changes in Site­Cat­a­lyst 15, mean­ing Site­Cat­a­lyst 14 and Site­cat­a­lyst 15 process data in an entirely dif­fer­ent way.

Let’s look at two myths that I hear a lot:

Site­Cat­a­lyst 14 is real-time”

That is part of the truth, but not the com­plete picture.

Data pro­cess­ing in Site­Cat­a­lyst fol­lows the prin­ci­ple “process as soon as you can”.

When a vis­i­tor calls up a web page, her browser sends a track­ing request. Site­Cat­a­lyst will almost imme­di­ately count this as a Page View. The visit, how­ever, is counted once it’s fin­ished, mean­ing after 30 min­utes of inactivity.

As a result, some of the reports in Site­Cat­a­lyst are prac­ti­cally real-time, the most pop­u­lar exam­ple being the Site Con­tent > Pages report. Cus­tom Traf­fic reports also dis­play real-time data for the Page Views metric.

Other reports are at least 30 min­utes behind, because the sys­tem will not process visit-based met­rics before the visit has actu­ally timed out. The same goes for some of the con­ver­sion reports and metrics.

Site­Cat­a­lyst 15 has a latency of 2 hours”

That is not accurate.

Site­Cat­a­lyst 15 processes data in batches. Those batches are usu­ally 60 min­utes of col­lected data. As an exam­ple, all track­ing data that arrives into Site­Cat­a­lyst between 11am and 11:59:59am is part of the same batch.

At 12:00, Site­Cat­a­lyst starts col­lect­ing data into a new batch. At the same time, the “11am batch” goes into pro­cess­ing. Pro­cess­ing of a 1-hour batch cur­rently takes around 30 minutes.

So, when do we see data?

If a track­ing request came in at 11:00:01am, it’ll be part of the “11am batch” which will fin­ish pro­cess­ing at around 12.30. The data will there­fore be vis­i­ble just under 90 min­utes after it happened.

If, on the other hand, the track­ing request came in at 11:59:59am, it’ll also be part of the “11am batch” and there­fore also vis­i­ble at 12.30, a mere 30 min­utes after it happened.

As a result of the batch approach, latency looks like this:

[Screenshot]

Latency in Site­Cat­a­lyst 15 — 60 ver­sus 30 minute slices

Notes

Not all data is processed in 60 minute batches. The batch size can be changed to 30 min­utes for crit­i­cal report suites (2 per customer).

Because a 30 minute batch can be processed in roughly 15 min­utes, the max­i­mum time for a hit to appear in Site­Cat­a­lyst is 45 min­utes, and the short­est time is 15 min­utes, fol­low­ing the same saw-tooth pat­tern as above.

What if you are fol­low­ing web activ­ity for a new release or after hav­ing sent a newslet­ter? What if your edi­to­r­ial team needs to know right now how that new arti­cle about that politi­cian performs?

Site­Cat­a­lyst 15 now has “Cur­rent Data” reports which work a lot like Site­Cat­a­lyst 14 worked: each report in Site­Cat­a­lyst 15 has a “Cur­rent Data” coun­ter­part which pro­vides the same latency that Site­cat­a­lyst 14 did.

[Screenshot]

Cur­rent Data Reports in Site­Cat­a­lyst 15

Those reports can be used to check data fo the cur­rent day or cur­rent day and the day before.

If you need Cur­rent Data reports, go to Admin > User Man­age­ment and add a user to the “Cur­rent Data” group. That user will now be able to see those reports.

[Screenshot]

Cur­rent Data Reports Group