Yesterday, my colleague, Adam Greco, wrote an outstanding post on the solution he debuted last week at Omniture Summit, which allows Twitter data to be pumped into SiteCatalyst, Discover, Data Warehouse, etc. using the Omniture Data Insertion API. Today I’d like to clarify the how-to of the solution so that you can get it up and running as smoothly as possible to begin reporting on your brand’s Twitter presence.

I won’t provide examples regarding the Data Insertion API framework, because of the wide variety in the types of environments where Twitter data can be captured and passed into SiteCatalyst; the developers at your organization can learn how to build this API using the resources at developer.omniture.com. Most importantly, keep in mind that the general implementation principles discussed below can be tweaked and altered according to your unique business needs, but the fundamentals regarding a.) the capture of this data from Twitter and b.) the compilation of this data into a SiteCatalyst image request are somewhat standard across all environments and implementations.

Adam used the example of a hypothetical web analyst at Comcast, who was asked by his CMO to provide data surrounding four key business questions that this solution can answer:

  1. How often is your company mentioned on Twitter?
  2. Is there ever a spike (positive or negative) in brand-related terms (in a week, day or even hourly)?
  3. Who are the people most often mentioning your company on social media tools and who are they communicating with the most?
  4. When are people on social media tools mentioning key product/service features that your Product Managers should know about?

Here is how you can obtain answers each of these questions using Twitter’s search API and Omniture’s Data Insertion API.

How often your company is mentioned on tools like Twitter?

Queries to Twitter’s search API (using http://search.twitter.com/search.atom?q=omniture, where “omniture” is replaced with a keyword of interest to your business) return XML which follows this general form:

<entry>
<id>tag:search.twitter.com,2005:1242044829</id>
<published>2009-02-23T19:58:41Z</published>
<link type="text/html" rel="alternate" href="http://twitter.com/jeffjordan/statuses/1242044829"/>
<title>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</title>
<content type="html">RT: Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a></content>
<updated>2009-02-23T19:58:41Z</updated>
<link type="image/png" rel="image" href="http://s3.amazonaws.com/twitter_production/profile_images/70442609/photo_normal.jpg"/>
<twitter:source><a href="http://www.twhirl.org/">twhirl</a></twitter:source>
<author>
<name>jeffjordan (jeffjordan)</name>
<uri>http://twitter.com/jeffjordan</uri>
</author>
</entry>

For each “tweet” (Twitter post/comment) returned by this query to the Twitter Search API, you would build and post an Omniture Data Insertion API request similar to this one:

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
</request>

where event1 is the event being used to count up the number of Twitter brand mentions. This Data Insertion API post, when performed for each result returned by the Twitter API, would produce a result like the screen shot given in Adam’s post:

Twitter data in SiteCatalyst

A key component of this solution is ensuring that we do not repeatedly count historical searches. If we make a query to the Twitter API once every 10 minutes, we only want to capture the number of searches that occurred during the last 10 minutes (i.e. since the last query). This can be done by capturing the last number in the <id> element (following the comma) in the top-most result returned by the Twitter API:

<id>tag:search.twitter.com,2005:1242044829</id>

and then putting that number into the “since_id=” parameter in your next Twitter API call, as follows:

http://search.twitter.com/search.atom?q=omniture&since_id=1242044829

Per the Twitter API documentation, this parameter “returns tweets with status ids greater than the given id.” In this example, This ensures that only the relevant tweets entered since the last API call are returned and entered into your Omniture Data Insertion API posts.

Is there ever a spike (positive or negative) in brand-related terms (in a week, day or even hourly)?

To break down the number of tweets, captured in event1 above, by the keywords being tweeted, simply pass the same value used in the q= parameter in your Twitter API call into an eVar in your Data Insertion API post. Storing the keyword in an eVar also allows you to perform multiple queries of the Twitter API to search for multiple keywords (e.g. if you have multiple products), if desired.

In the example above, the Twitter API query is http://search.twitter.com/search.atom?q=omniture. Here, you would simply pass “omniture” into an eVar in your Data Insertion API post, as follows. (This example uses eVar5 to store the keyword.)

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
</request>

You would then use SiteCatalyst Alerts, as Adam suggested, to report on trends in brand mentions on Twitter.

Who are the people most often mentioning your company on social media tools and who are they communicating with the most?

As described in Adam’s post, we will use two eVars to store the authors and recipients of tweets. This information is contained in the XML returned by the Twitter Search API, and obtaining it is simply a matter of parsing certain elements within the <entry> element.

Tweet author data can be obtained using the value of the <name> element within the <author> element in the XML example shown above. For your convenience, another example of these elements are reproduced below.

<author>
<name>OmnitureCare (Ben Gaines)</name>
<uri>http://twitter.com/omniturecare</uri>
</author>

The <name> element returns both the Twitter handle (e.g. OmnitureCare) as well as the friendly name (e.g. Ben Gaines). If you want to capture just the handle (or just the friendly name), you can use a function native to your development environment (such as substr() in PHP) to pull out the portion of the string that you need and place it into a variable on your server. For consistency’s sake, you may want to prepend “@” to the front of the Twitter handle captured in this manner (e.g. @OmnitureCare).

Tweet recipient data must be obtained using the <content> tag. Here, you will need to parse the string, looking for all substrings beginning with “@” and continuing until the next space character. For example, in the Twitter XML example above, the <content> element has a value of

RT: @Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a>

The recipient here is @Omni_man, so you would need to grab this value out of the <content> element and place it into a variable on your server. Adam suggested using “No recipient” as a value when there is no defined recipient, or (as in PHP) when strpos($content,"@") === FALSE, but you can use any value you want.

Once author and/or receipient have been captured, you can pass them into SiteCatalyst in the same Data Insertion API post that you are using to count a new mention of your brand, as in the example below. We will use eVar10 to store tweet author and eVar11 to store tweet recipient.

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
<evar10>@JeffJordan</evar10>
<evar11>@Omni_man</evar11>
</request>

This implementation will allow the kind of reporting shown in the screen shot below.

Tweet author and recipient

When are people on social media tools mentioning key product/service features that your Product Managers should know about?

This is actually the most straightforward of the implementation requirements given by the CMO in Adam’s example, because the full text of the Twitter post is available in the <content> element returned by the Twitter Search API. Once this text has been stripped of all HTML markup (note the “<” and “>” entities in the <content> element shown above), all you need to do is put the text into an eVar in your Data Insertion API post, as in the example below (using eVar12 for this purpose):

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
<evar10>@JeffJordan</evar10>
<evar11>@Omni_man</evar11>
<evar12>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</evar12>
</request>

While eVars are limited to 255 characters in length, Twitter limits tweets to 140 characters, so they should always be able to be passed into eVars. Having this full text available in SiteCatalyst will, as Adam explained, allow you to search for mentions of a particular product, feature, or service in connection with mentions of your company.

A few other important tips

  • Omniture recommends storing Twitter data in its own report suite. The reason for this is that the Data Insertion API posts described above will count page views, visits, and visitors, but mentions on Twitter are not the same as user interactions with your web site. It would be incorrect (and could wildly inflate traffic counts) to inject Twitter data into your production report suite.
  • Twitter currently does not charge for calls to its search API. Omniture does charge for server calls associated with passing Twitter data into SiteCatalyst.
  • Twitter’s API will return, at most, the 1,500 most recent searches mentioning the keyword(s) that you specify. Keep this in mind as you plan the “schedule” according to which you will be querying the API; if your brand or product is mentioned very frequently, you may need to run queries more frequently so that you do not miss results. Use the rpp= parameter to specify how many search results per “page” to return, and the page= parameter to specify what page to start from. These parameters allow you to sift through results when your brand is mentioned too often to return in a single query.
  • Make sure that the eVars storing the keyword, tweet author, tweet recipient, and tweet text are fully subrelated according to the breakdowns you will need to perform. You may not need to fully subrelate all four variables. If you want to break down Tweet Author by Tweet Recipient, but do not need to break down Tweet Recipient by Tweet Text, then you will need the Tweet Author eVar to be fully subrelated, but the Tweet Text eVar would need only basic subrelations. Omniture ClientCare can help you set up full subrelations on these eVars.
  • To obtain an accurate unique visitor (i.e. unique tweet author) count in the report suite storing your Twitter data, use a hash on the author handle (username) for the value in the <visitorid> element in your Data Insertion API posts.
  • Using an additional eVar, you can attempt to capture the “mood” of tweets regarding your brand by searching the <content> element for the appropriate emoticons, then translating them into a label (e.g. happy) which could be passed into the eVar. If the content of the tweet were “My Omniture reports look fantastic today! :)” you could detect the smiley face at the end of the tweet (using the strpos() function in PHP, for example), label this as “happy,” and pass that into an eVar at the time of the Data Insertion API post:

    <?xml version=1.0 encoding=UTF-8?>
    <request>
    <sc_xml_ver>1.0</sc_xml_ver>
    <reportsuiteid>yourrsid</reportsuiteid>
    <pageurl>http://www.yoursite.com</pageurl>
    <pagename>Twitter Mention</pagename>
    <events>event1</event>
    <evar5>omniture</evar5>
    <evar10>@OmnitureCare</evar10>
    <evar12>My Omniture reports look fantastic today! :)</evar12>
    <evar13>happy</evar13>
    </request>

    Although some sarcastic tweets will likely end up being labeled as “happy,” this can allow you to trend the attitudes in perception of your brand over time in SiteCatalyst.

Next time, I’ll discuss how you can use the SAINT API to capture Twitter data in a slightly different manner; it’s an alternative methodology with some real upside in certain cases.

18 comments
Leonel Volmer
Leonel Volmer

Hey there! Quick question that's entirely off topic. Do you know how to make your site mobile friendly? My site looks weird when viewing from my iphone 4. I'm trying to find a theme or plugin that might be able to resolve this issue. If you have any recommendations, please share. Thanks!

Jasmine Depoyster
Jasmine Depoyster

I think this is among the most important info for me. And i'm glad reading your article. But should remark on some general things, The site style is ideal, the articles is really nice : D. Good job, cheers

Berta Ducci
Berta Ducci

There are some good points in this blog however not sure if I agree with the entire article.

foretagsresor
foretagsresor

continue with the the good work on the site. I love it. Could maybe use some more updates more often, but i'm sure you got more or better stuff to do like we all have to do unfortunately. :)

Lavette Graetz
Lavette Graetz

Awesome website man, looks very nice. Finally found what I've been looking for, thanks!

Alexandra Friedman
Alexandra Friedman

Hi, nice post and am looking into integrating this into our Omniutre suite... Question: Is there a way I can see Tweets which link to Guidespot (the site I manage) if, let's say, the link in the Tweet text or any of the Tweet text for that matter, do not have any of my keywords or site name? Here is where this happens....Shortened urls... such as Bit.ly and tinyurl... Huge need and highly used on Twitter. Sometimes when people like something from the site they might not reference the site at all, so none of my keywords would work, and if they use a shortened URL, the name of my site would also not work... Is there something we can use to see that this Twitter user was a referrer to the site via whatever shortener they used... and without any keyword? With so many twitter apps and mobile users, finding the actual Tweets and brand advocates is getting harder. Long question, but hoping there is an easy answer? Thanks!

RRS_ATL
RRS_ATL

Oh my this looks interesting. I shall have to attempt this.

Aaron Bean
Aaron Bean

Does this integration track viral pass along or re-tweets? I'd like to be able to show the spread of a tweet and measure how far it has traveled/ its reach. There are other tools in the market from companies like Meteor Solutions but if Site Catalyst can do this too that would be sweet. Please let me know. Thanks.

Ben Gaines
Ben Gaines

Dev: The implementation methods described in this post (as well as in my subsequent posts on the SiteCatalyst-Twitter integration) will produce data that you will see in Omniture Discover. You shouldn't need to do anything differently to get the data into Discover.

Ben Gaines
Ben Gaines

Jennifer: This is a key question, and one that is difficult to address at this point. The problem is that there isn't a great way (yet) to tie a user's tweet to his/her experience on your site. I'll be posting shortly on how SiteCatalyst tracks visits and visitors, which is very closely tied to its ability to attribute revenue to previous user actions and variable values; without a way to tie this methodology (i.e., our use of visitor IDs) to a unique Twitter user, there isn't a reliable way to see revenue by twitter user. However, should there arise a better way of doing this, note that in my post located at http://is.gd/kT2B I describe a way to put Twitter data into the same report suite as your site data. Enjoy!

Dev
Dev

Hi Ben, This is really awesome! I'm pretty sure this can be integrated into the Discover product line as well, right? I guess it would be a much more extensive setup as opposed to this simplified approach. Is there any posting on how to integrate this into Discover?

Jennifer Yacenda
Jennifer Yacenda

Hi Ben, Thank you for the post! Sounds awesome! Quick question - in the above, you recommend storing Twitter data in its own report suite - how would we be able to correlate Twitter data with say a purchase event on our regular report suite? Thanks! Best, Jennifer

Ben Gaines
Ben Gaines

Kristi: Great question! Yes, you can do some things to isolate the user’s time zone. I’m still playing with it personally, because you’ll need to make an additional Twitter API call for each user to get the time zone, and Twitter limits you to 100 API calls per hour. Anyway, the time zone can be captured by making the following request: http://twitter.com/users/show/userid.xml EXAMPLE: http://twitter.com/users/show/omniturecare.xml In the result, you will see a <time_zone> element. You could grab its value for the author of each post and pass it into a SiteCatalyst variable. Please let me know if you have any additional questions about this!

Kristi Barrow
Kristi Barrow

Hi Ben, thanks for the post. Here's a question for you! How can we segment the data by country??? Is it possible to pull the IP address or the Twitter timezone for the user?? The reason being I am located in Australia (as are most of my clients), and any big name global brands (e.g. Coke, Expedia, Microsoft )would be completely swamped by US tweets vs Aussie tweets. This would make it very hard to obtain any meaningful data on Australian sentiment. Thoughts??? thanks Kristi

Ben Gaines
Ben Gaines

Jordan: Thanks for the positive feedback! I'm thrilled that you found the information useful. To answer your question: Yes, you can return results for multiple keywords and variations on keywords in a single Twitter API query. Just use the AND or OR operators to search for a whole list of terms. For example, http://search.twitter.com/search.atom?q=%22omniture+sitecatalyst%22+OR+clickmap would return results for “Omniture SiteCatalyst” (the entire phrase) as well as results for ClickMap.

Jordan
Jordan

What an awesome, awesome integration! This is just fantastic. Not knowing a lot about the Twitter API, can we search for multiple variations on terms in one hit to the Twitter API (such as "red" OR "black" OR "redblack" OR "red black" instead of only "red black", our actual company name)?

Ben Gaines
Ben Gaines

Alexandra: Check out my subsequent post, http://blogs.omniture.com/2009/02/25/a-couple-more-sitecatalyst-twitter-tips/. The best way to accomplish what you need is to add a campaign tracking code to the landing page URL before running it through a shortener. So, instead of inputting http://www.yoursite.com/landing_page.html into the shortener, you would input something like http://www.yoursite.com/landing_page.html?cid=123456. The shortener will preserve this tracking code. On the landing page, you can use the getQueryParam plug-in to capture the value, and you can know that the data associated with this tracking code represents traffic and conversion due to Twitter campaigns.

Ben Gaines
Ben Gaines

Aaron: Great point. Tracking re-tweets is a great piece of the Twitter measurement puzzle, and it could easily be done using the solution described in my post. You'd simply check the tweet text for "RT" and then set an additional custom event in the XML API post if that string is found in the tweet text. This would count total re-tweets and would let you view re-tweets by keyword/brand, Twitter user, etc. Let me know if you need more details or if that doesn't make sense.