Blog Post:Yesterday, my colleague, Adam Greco, wrote an outstanding post on the solution he debuted last week at Omniture Summit, which allows Twitter data to be pumped into SiteCatalyst, Discover, Data Warehouse, etc. using the Omniture Data Insertion API. Today I'd like to clarify the how-to of the solution so that you can get it up and running as smoothly as possible to begin reporting on your brand's Twitter presence. I won't provide examples regarding the Data Insertion API framework, because of the wide variety in the types of environments where Twitter data can be captured and passed into SiteCatalyst; the developers at your organization can learn how to build this API using the resources at developer.omniture.com. Most importantly, keep in mind that the general implementation principles discussed below can be tweaked and altered according to your unique business needs, but the fundamentals regarding a.) the capture of this data from Twitter and b.) the compilation of this data into a SiteCatalyst image request are somewhat standard across all environments and implementations. Adam used the example of a hypothetical web analyst at Comcast, who was asked by his CMO to provide data surrounding four key business questions that this solution can answer:
  1. How often is your company mentioned on Twitter?
  2. Is there ever a spike (positive or negative) in brand-related terms (in a week, day or even hourly)?
  3. Who are the people most often mentioning your company on social media tools and who are they communicating with the most?
  4. When are people on social media tools mentioning key product/service features that your Product Managers should know about?
Here is how you can obtain answers each of these questions using Twitter's search API and Omniture's Data Insertion API. How often your company is mentioned on tools like Twitter? Queries to Twitter's search API (using http://search.twitter.com/search.atom?q=omniture, where "omniture" is replaced with a keyword of interest to your business) return XML which follows this general form: <entry> <id>tag:search.twitter.com,2005:1242044829</id> <published>2009-02-23T19:58:41Z</published> <link type="text/html" rel="alternate" href="http://twitter.com/jeffjordan/statuses/1242044829"/> <title>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</title> <content type="html">RT: Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a></content> <updated>2009-02-23T19:58:41Z</updated> <link type="image/png" rel="image" href="http://s3.amazonaws.com/twitter_production/profile_images/70442609/photo_normal.jpg"/> <twitter:source><a href="http://www.twhirl.org/">twhirl</a></twitter:source> <author> <name>jeffjordan (jeffjordan)</name> <uri>http://twitter.com/jeffjordan</uri> </author> </entry> For each "tweet" (Twitter post/comment) returned by this query to the Twitter Search API, you would build and post an Omniture Data Insertion API request similar to this one: <?xml version=1.0 encoding=UTF-8?> <request> <sc_xml_ver>1.0</sc_xml_ver> <reportsuiteid>yourrsid</reportsuiteid> <pageurl>http://www.yoursite.com</pageurl> <pagename>Twitter Mention</pagename> <events>event1</event> </request> where event1 is the event being used to count up the number of Twitter brand mentions. This Data Insertion API post, when performed for each result returned by the Twitter API, would produce a result like the screen shot given in Adam's post: Twitter data in SiteCatalyst A key component of this solution is ensuring that we do not repeatedly count historical searches. If we make a query to the Twitter API once every 10 minutes, we only want to capture the number of searches that occurred during the last 10 minutes (i.e. since the last query). This can be done by capturing the last number in the <id> element (following the comma) in the top-most result returned by the Twitter API: <id>tag:search.twitter.com,2005:1242044829</id> and then putting that number into the "since_id=" parameter in your next Twitter API call, as follows: http://search.twitter.com/search.atom?q=omniture&since_id=1242044829 Per the Twitter API documentation, this parameter "returns tweets with status ids greater than the given id." In this example, This ensures that only the relevant tweets entered since the last API call are returned and entered into your Omniture Data Insertion API posts. Is there ever a spike (positive or negative) in brand-related terms (in a week, day or even hourly)? To break down the number of tweets, captured in event1 above, by the keywords being tweeted, simply pass the same value used in the q= parameter in your Twitter API call into an eVar in your Data Insertion API post. Storing the keyword in an eVar also allows you to perform multiple queries of the Twitter API to search for multiple keywords (e.g. if you have multiple products), if desired. In the example above, the Twitter API query is http://search.twitter.com/search.atom?q=omniture. Here, you would simply pass "omniture" into an eVar in your Data Insertion API post, as follows. (This example uses eVar5 to store the keyword.) <?xml version=1.0 encoding=UTF-8?> <request> <sc_xml_ver>1.0</sc_xml_ver> <reportsuiteid>yourrsid</reportsuiteid> <pageurl>http://www.yoursite.com</pageurl> <pagename>Twitter Mention</pagename> <events>event1</event> <evar5>omniture</evar5> </request> You would then use SiteCatalyst Alerts, as Adam suggested, to report on trends in brand mentions on Twitter. Who are the people most often mentioning your company on social media tools and who are they communicating with the most? As described in Adam's post, we will use two eVars to store the authors and recipients of tweets. This information is contained in the XML returned by the Twitter Search API, and obtaining it is simply a matter of parsing certain elements within the <entry> element. Tweet author data can be obtained using the value of the <name> element within the <author> element in the XML example shown above. For your convenience, another example of these elements are reproduced below. <author> <name>OmnitureCare (Ben Gaines)</name> <uri>http://twitter.com/omniturecare</uri> </author> The <name> element returns both the Twitter handle (e.g. OmnitureCare) as well as the friendly name (e.g. Ben Gaines). If you want to capture just the handle (or just the friendly name), you can use a function native to your development environment (such as substr() in PHP) to pull out the portion of the string that you need and place it into a variable on your server. For consistency's sake, you may want to prepend "@" to the front of the Twitter handle captured in this manner (e.g. @OmnitureCare). Tweet recipient data must be obtained using the <content> tag. Here, you will need to parse the string, looking for all substrings beginning with "@" and continuing until the next space character. For example, in the Twitter XML example above, the <content> element has a value of RT: @Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a> The recipient here is @Omni_man, so you would need to grab this value out of the <content> element and place it into a variable on your server. Adam suggested using "No recipient" as a value when there is no defined recipient, or (as in PHP) when strpos($content,"@") === FALSE, but you can use any value you want. Once author and/or receipient have been captured, you can pass them into SiteCatalyst in the same Data Insertion API post that you are using to count a new mention of your brand, as in the example below. We will use eVar10 to store tweet author and eVar11 to store tweet recipient. <?xml version=1.0 encoding=UTF-8?> <request> <sc_xml_ver>1.0</sc_xml_ver> <reportsuiteid>yourrsid</reportsuiteid> <pageurl>http://www.yoursite.com</pageurl> <pagename>Twitter Mention</pagename> <events>event1</event> <evar5>omniture</evar5> <evar10>@JeffJordan</evar10> <evar11>@Omni_man</evar11> </request> This implementation will allow the kind of reporting shown in the screen shot below. Tweet author and recipient When are people on social media tools mentioning key product/service features that your Product Managers should know about? This is actually the most straightforward of the implementation requirements given by the CMO in Adam's example, because the full text of the Twitter post is available in the <content> element returned by the Twitter Search API. Once this text has been stripped of all HTML markup (note the "<" and ">" entities in the <content> element shown above), all you need to do is put the text into an eVar in your Data Insertion API post, as in the example below (using eVar12 for this purpose): <?xml version=1.0 encoding=UTF-8?> <request> <sc_xml_ver>1.0</sc_xml_ver> <reportsuiteid>yourrsid</reportsuiteid> <pageurl>http://www.yoursite.com</pageurl> <pagename>Twitter Mention</pagename> <events>event1</event> <evar5>omniture</evar5> <evar10>@JeffJordan</evar10> <evar11>@Omni_man</evar11> <evar12>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</evar12> </request> While eVars are limited to 255 characters in length, Twitter limits tweets to 140 characters, so they should always be able to be passed into eVars. Having this full text available in SiteCatalyst will, as Adam explained, allow you to search for mentions of a particular product, feature, or service in connection with mentions of your company. A few other important tips Next time, I'll discuss how you can use the SAINT API to capture Twitter data in a slightly different manner; it's an alternative methodology with some real upside in certain cases.
Author: Date Created:February 24, 2009 Date Published: Headline:Implementing Twitter Data Tracking in Omniture SiteCatalyst Social Counts: Keywords: Publisher:Adobe Image:https://blogs.adobe.com/digitalmarketing/wp-content/uploads/no-image/no-image.jpg

Yesterday, my colleague, Adam Greco, wrote an outstanding post on the solution he debuted last week at Omniture Summit, which allows Twitter data to be pumped into SiteCatalyst, Discover, Data Warehouse, etc. using the Omniture Data Insertion API. Today I’d like to clarify the how-to of the solution so that you can get it up and running as smoothly as possible to begin reporting on your brand’s Twitter presence.

I won’t provide examples regarding the Data Insertion API framework, because of the wide variety in the types of environments where Twitter data can be captured and passed into SiteCatalyst; the developers at your organization can learn how to build this API using the resources at developer.omniture.com. Most importantly, keep in mind that the general implementation principles discussed below can be tweaked and altered according to your unique business needs, but the fundamentals regarding a.) the capture of this data from Twitter and b.) the compilation of this data into a SiteCatalyst image request are somewhat standard across all environments and implementations.

Adam used the example of a hypothetical web analyst at Comcast, who was asked by his CMO to provide data surrounding four key business questions that this solution can answer:

  1. How often is your company mentioned on Twitter?
  2. Is there ever a spike (positive or negative) in brand-related terms (in a week, day or even hourly)?
  3. Who are the people most often mentioning your company on social media tools and who are they communicating with the most?
  4. When are people on social media tools mentioning key product/service features that your Product Managers should know about?

Here is how you can obtain answers each of these questions using Twitter’s search API and Omniture’s Data Insertion API.

How often your company is mentioned on tools like Twitter?

Queries to Twitter’s search API (using http://search.twitter.com/search.atom?q=omniture, where “omniture” is replaced with a keyword of interest to your business) return XML which follows this general form:

<entry>
<id>tag:search.twitter.com,2005:1242044829</id>
<published>2009-02-23T19:58:41Z</published>
<link type="text/html" rel="alternate" href="http://twitter.com/jeffjordan/statuses/1242044829"/>
<title>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</title>
<content type="html">RT: Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a></content>
<updated>2009-02-23T19:58:41Z</updated>
<link type="image/png" rel="image" href="http://s3.amazonaws.com/twitter_production/profile_images/70442609/photo_normal.jpg"/>
<twitter:source><a href="http://www.twhirl.org/">twhirl</a></twitter:source>
<author>
<name>jeffjordan (jeffjordan)</name>
<uri>http://twitter.com/jeffjordan</uri>
</author>
</entry>

For each “tweet” (Twitter post/comment) returned by this query to the Twitter Search API, you would build and post an Omniture Data Insertion API request similar to this one:

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
</request>

where event1 is the event being used to count up the number of Twitter brand mentions. This Data Insertion API post, when performed for each result returned by the Twitter API, would produce a result like the screen shot given in Adam’s post:

Twitter data in SiteCatalyst

A key component of this solution is ensuring that we do not repeatedly count historical searches. If we make a query to the Twitter API once every 10 minutes, we only want to capture the number of searches that occurred during the last 10 minutes (i.e. since the last query). This can be done by capturing the last number in the <id> element (following the comma) in the top-most result returned by the Twitter API:

<id>tag:search.twitter.com,2005:1242044829</id>

and then putting that number into the “since_id=” parameter in your next Twitter API call, as follows:

http://search.twitter.com/search.atom?q=omniture&since_id=1242044829

Per the Twitter API documentation, this parameter “returns tweets with status ids greater than the given id.” In this example, This ensures that only the relevant tweets entered since the last API call are returned and entered into your Omniture Data Insertion API posts.

Is there ever a spike (positive or negative) in brand-related terms (in a week, day or even hourly)?

To break down the number of tweets, captured in event1 above, by the keywords being tweeted, simply pass the same value used in the q= parameter in your Twitter API call into an eVar in your Data Insertion API post. Storing the keyword in an eVar also allows you to perform multiple queries of the Twitter API to search for multiple keywords (e.g. if you have multiple products), if desired.

In the example above, the Twitter API query is http://search.twitter.com/search.atom?q=omniture. Here, you would simply pass “omniture” into an eVar in your Data Insertion API post, as follows. (This example uses eVar5 to store the keyword.)

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
</request>

You would then use SiteCatalyst Alerts, as Adam suggested, to report on trends in brand mentions on Twitter.

Who are the people most often mentioning your company on social media tools and who are they communicating with the most?

As described in Adam’s post, we will use two eVars to store the authors and recipients of tweets. This information is contained in the XML returned by the Twitter Search API, and obtaining it is simply a matter of parsing certain elements within the <entry> element.

Tweet author data can be obtained using the value of the <name> element within the <author> element in the XML example shown above. For your convenience, another example of these elements are reproduced below.

<author>
<name>OmnitureCare (Ben Gaines)</name>
<uri>http://twitter.com/omniturecare</uri>
</author>

The <name> element returns both the Twitter handle (e.g. OmnitureCare) as well as the friendly name (e.g. Ben Gaines). If you want to capture just the handle (or just the friendly name), you can use a function native to your development environment (such as substr() in PHP) to pull out the portion of the string that you need and place it into a variable on your server. For consistency’s sake, you may want to prepend “@” to the front of the Twitter handle captured in this manner (e.g. @OmnitureCare).

Tweet recipient data must be obtained using the <content> tag. Here, you will need to parse the string, looking for all substrings beginning with “@” and continuing until the next space character. For example, in the Twitter XML example above, the <content> element has a value of

RT: @Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a>

The recipient here is @Omni_man, so you would need to grab this value out of the <content> element and place it into a variable on your server. Adam suggested using “No recipient” as a value when there is no defined recipient, or (as in PHP) when strpos($content,"@") === FALSE, but you can use any value you want.

Once author and/or receipient have been captured, you can pass them into SiteCatalyst in the same Data Insertion API post that you are using to count a new mention of your brand, as in the example below. We will use eVar10 to store tweet author and eVar11 to store tweet recipient.

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
<evar10>@JeffJordan</evar10>
<evar11>@Omni_man</evar11>
</request>

This implementation will allow the kind of reporting shown in the screen shot below.

Tweet author and recipient

When are people on social media tools mentioning key product/service features that your Product Managers should know about?

This is actually the most straightforward of the implementation requirements given by the CMO in Adam’s example, because the full text of the Twitter post is available in the <content> element returned by the Twitter Search API. Once this text has been stripped of all HTML markup (note the “<” and “>” entities in the <content> element shown above), all you need to do is put the text into an eVar in your Data Insertion API post, as in the example below (using eVar12 for this purpose):

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
<evar10>@JeffJordan</evar10>
<evar11>@Omni_man</evar11>
<evar12>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</evar12>
</request>

While eVars are limited to 255 characters in length, Twitter limits tweets to 140 characters, so they should always be able to be passed into eVars. Having this full text available in SiteCatalyst will, as Adam explained, allow you to search for mentions of a particular product, feature, or service in connection with mentions of your company.

A few other important tips

  • Omniture recommends storing Twitter data in its own report suite. The reason for this is that the Data Insertion API posts described above will count page views, visits, and visitors, but mentions on Twitter are not the same as user interactions with your web site. It would be incorrect (and could wildly inflate traffic counts) to inject Twitter data into your production report suite.
  • Twitter currently does not charge for calls to its search API. Omniture does charge for server calls associated with passing Twitter data into SiteCatalyst.
  • Twitter’s API will return, at most, the 1,500 most recent searches mentioning the keyword(s) that you specify. Keep this in mind as you plan the “schedule” according to which you will be querying the API; if your brand or product is mentioned very frequently, you may need to run queries more frequently so that you do not miss results. Use the rpp= parameter to specify how many search results per “page” to return, and the page= parameter to specify what page to start from. These parameters allow you to sift through results when your brand is mentioned too often to return in a single query.
  • Make sure that the eVars storing the keyword, tweet author, tweet recipient, and tweet text are fully subrelated according to the breakdowns you will need to perform. You may not need to fully subrelate all four variables. If you want to break down Tweet Author by Tweet Recipient, but do not need to break down Tweet Recipient by Tweet Text, then you will need the Tweet Author eVar to be fully subrelated, but the Tweet Text eVar would need only basic subrelations. Omniture ClientCare can help you set up full subrelations on these eVars.
  • To obtain an accurate unique visitor (i.e. unique tweet author) count in the report suite storing your Twitter data, use a hash on the author handle (username) for the value in the <visitorid> element in your Data Insertion API posts.
  • Using an additional eVar, you can attempt to capture the “mood” of tweets regarding your brand by searching the <content> element for the appropriate emoticons, then translating them into a label (e.g. happy) which could be passed into the eVar. If the content of the tweet were “My Omniture reports look fantastic today! :)” you could detect the smiley face at the end of the tweet (using the strpos() function in PHP, for example), label this as “happy,” and pass that into an eVar at the time of the Data Insertion API post:

    <?xml version=1.0 encoding=UTF-8?>
    <request>
    <sc_xml_ver>1.0</sc_xml_ver>
    <reportsuiteid>yourrsid</reportsuiteid>
    <pageurl>http://www.yoursite.com</pageurl>
    <pagename>Twitter Mention</pagename>
    <events>event1</event>
    <evar5>omniture</evar5>
    <evar10>@OmnitureCare</evar10>
    <evar12>My Omniture reports look fantastic today! :)</evar12>
    <evar13>happy</evar13>
    </request>

    Although some sarcastic tweets will likely end up being labeled as “happy,” this can allow you to trend the attitudes in perception of your brand over time in SiteCatalyst.

Next time, I’ll discuss how you can use the SAINT API to capture Twitter data in a slightly different manner; it’s an alternative methodology with some real upside in certain cases.