Yes­ter­day, my col­league, Adam Greco, wrote an out­stand­ing post on the solu­tion he debuted last week at Omni­ture Sum­mit, which allows Twit­ter data to be pumped into Site­Cat­a­lyst, Dis­cover, Data Ware­house, etc. using the Omni­ture Data Inser­tion API. Today I’d like to clar­ify the how-to of the solu­tion so that you can get it up and run­ning as smoothly as pos­si­ble to begin report­ing on your brand’s Twit­ter presence.

I won’t pro­vide exam­ples regard­ing the Data Inser­tion API frame­work, because of the wide vari­ety in the types of envi­ron­ments where Twit­ter data can be cap­tured and passed into Site­Cat­a­lyst; the devel­op­ers at your orga­ni­za­tion can learn how to build this API using the resources at devel​oper​.omni​ture​.com. Most impor­tantly, keep in mind that the gen­eral imple­men­ta­tion prin­ci­ples dis­cussed below can be tweaked and altered accord­ing to your unique busi­ness needs, but the fun­da­men­tals regard­ing a.) the cap­ture of this data from Twit­ter and b.) the com­pi­la­tion of this data into a Site­Cat­a­lyst image request are some­what stan­dard across all envi­ron­ments and implementations.

Adam used the exam­ple of a hypo­thet­i­cal web ana­lyst at Com­cast, who was asked by his CMO to pro­vide data sur­round­ing four key busi­ness ques­tions that this solu­tion can answer:

  1. How often is your com­pany men­tioned on Twitter?
  2. Is there ever a spike (pos­i­tive or neg­a­tive) in brand-related terms (in a week, day or even hourly)?
  3. Who are the peo­ple most often men­tion­ing your com­pany on social media tools and who are they com­mu­ni­cat­ing with the most?
  4. When are peo­ple on social media tools men­tion­ing key product/service fea­tures that your Prod­uct Man­agers should know about?

Here is how you can obtain answers each of these ques­tions using Twitter’s search API and Omniture’s Data Inser­tion API.

How often your com­pany is men­tioned on tools like Twitter?

Queries to Twitter’s search API (using http://search.twitter.com/search.atom?q=omniture, where “omni­ture” is replaced with a key­word of inter­est to your busi­ness) return XML which fol­lows this gen­eral form:

<entry>
<id>tag:search.twitter.com,2005:1242044829</id>
<published>2009-02-23T19:58:41Z</published>
<link type="text/html" rel="alternate" href="http://twitter.com/jeffjordan/statuses/1242044829"/>
<title>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</title>
<content type="html">RT: Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a></content>
<updated>2009-02-23T19:58:41Z</updated>
<link type="image/png" rel="image" href="http://s3.amazonaws.com/twitter_production/profile_images/70442609/photo_normal.jpg"/>
<twitter:source><a href="http://www.twhirl.org/">twhirl</a></twitter:source>
<author>
<name>jeffjordan (jeffjordan)</name>
<uri>http://twitter.com/jeffjordan</uri>
</author>
</entry>

For each “tweet” (Twit­ter post/comment) returned by this query to the Twit­ter Search API, you would build and post an Omni­ture Data Inser­tion API request sim­i­lar to this one:

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
</request>

where event1 is the event being used to count up the num­ber of Twit­ter brand men­tions. This Data Inser­tion API post, when per­formed for each result returned by the Twit­ter API, would pro­duce a result like the screen shot given in Adam’s post:

Twitter data in SiteCatalyst

A key com­po­nent of this solu­tion is ensur­ing that we do not repeat­edly count his­tor­i­cal searches. If we make a query to the Twit­ter API once every 10 min­utes, we only want to cap­ture the num­ber of searches that occurred dur­ing the last 10 min­utes (i.e. since the last query). This can be done by cap­tur­ing the last num­ber in the <id> ele­ment (fol­low­ing the comma) in the top-most result returned by the Twit­ter API:

<id>tag:search.twitter.com,2005:1242044829</id>

and then putting that num­ber into the “since_id=” para­me­ter in your next Twit­ter API call, as follows:

http://search.twitter.com/search.atom?q=omniture&since_id=1242044829

Per the Twit­ter API doc­u­men­ta­tion, this para­me­ter “returns tweets with sta­tus ids greater than the given id.” In this exam­ple, This ensures that only the rel­e­vant tweets entered since the last API call are returned and entered into your Omni­ture Data Inser­tion API posts.

Is there ever a spike (pos­i­tive or neg­a­tive) in brand-related terms (in a week, day or even hourly)?

To break down the num­ber of tweets, cap­tured in event1 above, by the key­words being tweeted, sim­ply pass the same value used in the q= para­me­ter in your Twit­ter API call into an eVar in your Data Inser­tion API post. Stor­ing the key­word in an eVar also allows you to per­form mul­ti­ple queries of the Twit­ter API to search for mul­ti­ple key­words (e.g. if you have mul­ti­ple prod­ucts), if desired.

In the exam­ple above, the Twit­ter API query is http://search.twitter.com/search.atom?q=omniture. Here, you would sim­ply pass “omni­ture” into an eVar in your Data Inser­tion API post, as fol­lows. (This exam­ple uses eVar5 to store the keyword.)

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
</request>

You would then use Site­Cat­a­lyst Alerts, as Adam sug­gested, to report on trends in brand men­tions on Twitter.

Who are the peo­ple most often men­tion­ing your com­pany on social media tools and who are they com­mu­ni­cat­ing with the most?

As described in Adam’s post, we will use two eVars to store the authors and recip­i­ents of tweets. This infor­ma­tion is con­tained in the XML returned by the Twit­ter Search API, and obtain­ing it is sim­ply a mat­ter of pars­ing cer­tain ele­ments within the <entry> element.

Tweet author data can be obtained using the value of the <name> ele­ment within the <author> ele­ment in the XML exam­ple shown above. For your con­ve­nience, another exam­ple of these ele­ments are repro­duced below.

<author>
<name>OmnitureCare (Ben Gaines)</name>
<uri>http://twitter.com/omniturecare</uri>
</author>

The <name> ele­ment returns both the Twit­ter han­dle (e.g. Omni­ture­Care) as well as the friendly name (e.g. Ben Gaines). If you want to cap­ture just the han­dle (or just the friendly name), you can use a func­tion native to your devel­op­ment envi­ron­ment (such as substr() in PHP) to pull out the por­tion of the string that you need and place it into a vari­able on your server. For consistency’s sake, you may want to prepend “@” to the front of the Twit­ter han­dle cap­tured in this man­ner (e.g. @OmnitureCare).

Tweet recip­i­ent data must be obtained using the <con­tent> tag. Here, you will need to parse the string, look­ing for all sub­strings begin­ning with “@” and con­tin­u­ing until the next space char­ac­ter. For exam­ple, in the Twit­ter XML exam­ple above, the <con­tent> ele­ment has a value of

RT: @Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a>

The recip­i­ent here is @Omni_man, so you would need to grab this value out of the <con­tent> ele­ment and place it into a vari­able on your server. Adam sug­gested using “No recip­i­ent” as a value when there is no defined recip­i­ent, or (as in PHP) when strpos($content,"@") === FALSE, but you can use any value you want.

Once author and/or receip­i­ent have been cap­tured, you can pass them into Site­Cat­a­lyst in the same Data Inser­tion API post that you are using to count a new men­tion of your brand, as in the exam­ple below. We will use eVar10 to store tweet author and eVar11 to store tweet recipient.

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
<evar10>@JeffJordan</evar10>
<evar11>@Omni_man</evar11>
</request>

This imple­men­ta­tion will allow the kind of report­ing shown in the screen shot below.

Tweet author and recipient

When are peo­ple on social media tools men­tion­ing key product/service fea­tures that your Prod­uct Man­agers should know about?

This is actu­ally the most straight­for­ward of the imple­men­ta­tion require­ments given by the CMO in Adam’s exam­ple, because the full text of the Twit­ter post is avail­able in the <con­tent> ele­ment returned by the Twit­ter Search API. Once this text has been stripped of all HTML markup (note the “<” and “>” enti­ties in the <con­tent> ele­ment shown above), all you need to do is put the text into an eVar in your Data Inser­tion API post, as in the exam­ple below (using eVar12 for this purpose):

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
<evar10>@JeffJordan</evar10>
<evar11>@Omni_man</evar11>
<evar12>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</evar12>
</request>

While eVars are lim­ited to 255 char­ac­ters in length, Twit­ter lim­its tweets to 140 char­ac­ters, so they should always be able to be passed into eVars. Hav­ing this full text avail­able in Site­Cat­a­lyst will, as Adam explained, allow you to search for men­tions of a par­tic­u­lar prod­uct, fea­ture, or ser­vice in con­nec­tion with men­tions of your company.

A few other impor­tant tips

  • Omni­ture rec­om­mends stor­ing Twit­ter data in its own report suite. The rea­son for this is that the Data Inser­tion API posts described above will count page views, vis­its, and vis­i­tors, but men­tions on Twit­ter are not the same as user inter­ac­tions with your web site. It would be incor­rect (and could wildly inflate traf­fic counts) to inject Twit­ter data into your pro­duc­tion report suite.
  • Twit­ter cur­rently does not charge for calls to its search API. Omni­ture does charge for server calls asso­ci­ated with pass­ing Twit­ter data into SiteCatalyst.
  • Twitter’s API will return, at most, the 1,500 most recent searches men­tion­ing the keyword(s) that you spec­ify. Keep this in mind as you plan the “sched­ule” accord­ing to which you will be query­ing the API; if your brand or prod­uct is men­tioned very fre­quently, you may need to run queries more fre­quently so that you do not miss results. Use the rpp= para­me­ter to spec­ify how many search results per “page” to return, and the page= para­me­ter to spec­ify what page to start from. These para­me­ters allow you to sift through results when your brand is men­tioned too often to return in a sin­gle query.
  • Make sure that the eVars stor­ing the key­word, tweet author, tweet recip­i­ent, and tweet text are fully sub­re­lated accord­ing to the break­downs you will need to per­form. You may not need to fully sub­re­late all four vari­ables. If you want to break down Tweet Author by Tweet Recip­i­ent, but do not need to break down Tweet Recip­i­ent by Tweet Text, then you will need the Tweet Author eVar to be fully sub­re­lated, but the Tweet Text eVar would need only basic sub­re­la­tions. Omni­ture Client­Care can help you set up full sub­re­la­tions on these eVars.
  • To obtain an accu­rate unique vis­i­tor (i.e. unique tweet author) count in the report suite stor­ing your Twit­ter data, use a hash on the author han­dle (user­name) for the value in the <vis­i­torid> ele­ment in your Data Inser­tion API posts.
  • Using an addi­tional eVar, you can attempt to cap­ture the “mood” of tweets regard­ing your brand by search­ing the <con­tent> ele­ment for the appro­pri­ate emoti­cons, then trans­lat­ing them into a label (e.g. happy) which could be passed into the eVar. If the con­tent of the tweet were “My Omni­ture reports look fan­tas­tic today! :)” you could detect the smi­ley face at the end of the tweet (using the strpos() func­tion in PHP, for exam­ple), label this as “happy,” and pass that into an eVar at the time of the Data Inser­tion API post:

    <?xml version=1.0 encoding=UTF-8?>
    <request>
    <sc_xml_ver>1.0</sc_xml_ver>
    <reportsuiteid>yourrsid</reportsuiteid>
    <pageurl>http://www.yoursite.com</pageurl>
    <pagename>Twitter Mention</pagename>
    <events>event1</event>
    <evar5>omniture</evar5>
    <evar10>@OmnitureCare</evar10>
    <evar12>My Omniture reports look fantastic today! :)</evar12>
    <evar13>happy</evar13>
    </request>

    Although some sar­cas­tic tweets will likely end up being labeled as “happy,” this can allow you to trend the atti­tudes in per­cep­tion of your brand over time in SiteCatalyst.

Next time, I’ll dis­cuss how you can use the SAINT API to cap­ture Twit­ter data in a slightly dif­fer­ent man­ner; it’s an alter­na­tive method­ol­ogy with some real upside in cer­tain cases.

18 comments
Leonel Volmer
Leonel Volmer

Hey there! Quick question that's entirely off topic. Do you know how to make your site mobile friendly? My site looks weird when viewing from my iphone 4. I'm trying to find a theme or plugin that might be able to resolve this issue. If you have any recommendations, please share. Thanks!

Jasmine Depoyster
Jasmine Depoyster

I think this is among the most important info for me. And i'm glad reading your article. But should remark on some general things, The site style is ideal, the articles is really nice : D. Good job, cheers

Berta Ducci
Berta Ducci

There are some good points in this blog however not sure if I agree with the entire article.

foretagsresor
foretagsresor

continue with the the good work on the site. I love it. Could maybe use some more updates more often, but i'm sure you got more or better stuff to do like we all have to do unfortunately. :)

Lavette Graetz
Lavette Graetz

Awesome website man, looks very nice. Finally found what I've been looking for, thanks!

Alexandra Friedman
Alexandra Friedman

Hi, nice post and am looking into integrating this into our Omniutre suite... Question: Is there a way I can see Tweets which link to Guidespot (the site I manage) if, let's say, the link in the Tweet text or any of the Tweet text for that matter, do not have any of my keywords or site name? Here is where this happens....Shortened urls... such as Bit.ly and tinyurl... Huge need and highly used on Twitter. Sometimes when people like something from the site they might not reference the site at all, so none of my keywords would work, and if they use a shortened URL, the name of my site would also not work... Is there something we can use to see that this Twitter user was a referrer to the site via whatever shortener they used... and without any keyword? With so many twitter apps and mobile users, finding the actual Tweets and brand advocates is getting harder. Long question, but hoping there is an easy answer? Thanks!

RRS_ATL
RRS_ATL

Oh my this looks interesting. I shall have to attempt this.

Aaron Bean
Aaron Bean

Does this integration track viral pass along or re-tweets? I'd like to be able to show the spread of a tweet and measure how far it has traveled/ its reach. There are other tools in the market from companies like Meteor Solutions but if Site Catalyst can do this too that would be sweet. Please let me know. Thanks.

Ben Gaines
Ben Gaines

Dev: The implementation methods described in this post (as well as in my subsequent posts on the SiteCatalyst-Twitter integration) will produce data that you will see in Omniture Discover. You shouldn't need to do anything differently to get the data into Discover.

Ben Gaines
Ben Gaines

Jennifer: This is a key question, and one that is difficult to address at this point. The problem is that there isn't a great way (yet) to tie a user's tweet to his/her experience on your site. I'll be posting shortly on how SiteCatalyst tracks visits and visitors, which is very closely tied to its ability to attribute revenue to previous user actions and variable values; without a way to tie this methodology (i.e., our use of visitor IDs) to a unique Twitter user, there isn't a reliable way to see revenue by twitter user. However, should there arise a better way of doing this, note that in my post located at http://is.gd/kT2B I describe a way to put Twitter data into the same report suite as your site data. Enjoy!

Dev
Dev

Hi Ben, This is really awesome! I'm pretty sure this can be integrated into the Discover product line as well, right? I guess it would be a much more extensive setup as opposed to this simplified approach. Is there any posting on how to integrate this into Discover?

Jennifer Yacenda
Jennifer Yacenda

Hi Ben, Thank you for the post! Sounds awesome! Quick question - in the above, you recommend storing Twitter data in its own report suite - how would we be able to correlate Twitter data with say a purchase event on our regular report suite? Thanks! Best, Jennifer

Ben Gaines
Ben Gaines

Kristi: Great question! Yes, you can do some things to isolate the user’s time zone. I’m still playing with it personally, because you’ll need to make an additional Twitter API call for each user to get the time zone, and Twitter limits you to 100 API calls per hour. Anyway, the time zone can be captured by making the following request: http://twitter.com/users/show/userid.xml EXAMPLE: http://twitter.com/users/show/omniturecare.xml In the result, you will see a <time_zone> element. You could grab its value for the author of each post and pass it into a SiteCatalyst variable. Please let me know if you have any additional questions about this!

Kristi Barrow
Kristi Barrow

Hi Ben, thanks for the post. Here's a question for you! How can we segment the data by country??? Is it possible to pull the IP address or the Twitter timezone for the user?? The reason being I am located in Australia (as are most of my clients), and any big name global brands (e.g. Coke, Expedia, Microsoft )would be completely swamped by US tweets vs Aussie tweets. This would make it very hard to obtain any meaningful data on Australian sentiment. Thoughts??? thanks Kristi

Ben Gaines
Ben Gaines

Jordan: Thanks for the positive feedback! I'm thrilled that you found the information useful. To answer your question: Yes, you can return results for multiple keywords and variations on keywords in a single Twitter API query. Just use the AND or OR operators to search for a whole list of terms. For example, http://search.twitter.com/search.atom?q=%22omniture+sitecatalyst%22+OR+clickmap would return results for “Omniture SiteCatalyst” (the entire phrase) as well as results for ClickMap.

Jordan
Jordan

What an awesome, awesome integration! This is just fantastic. Not knowing a lot about the Twitter API, can we search for multiple variations on terms in one hit to the Twitter API (such as "red" OR "black" OR "redblack" OR "red black" instead of only "red black", our actual company name)?

Ben Gaines
Ben Gaines

Alexandra: Check out my subsequent post, http://blogs.omniture.com/2009/02/25/a-couple-more-sitecatalyst-twitter-tips/. The best way to accomplish what you need is to add a campaign tracking code to the landing page URL before running it through a shortener. So, instead of inputting http://www.yoursite.com/landing_page.html into the shortener, you would input something like http://www.yoursite.com/landing_page.html?cid=123456. The shortener will preserve this tracking code. On the landing page, you can use the getQueryParam plug-in to capture the value, and you can know that the data associated with this tracking code represents traffic and conversion due to Twitter campaigns.

Ben Gaines
Ben Gaines

Aaron: Great point. Tracking re-tweets is a great piece of the Twitter measurement puzzle, and it could easily be done using the solution described in my post. You'd simply check the tweet text for "RT" and then set an additional custom event in the XML API post if that string is found in the tweet text. This would count total re-tweets and would let you view re-tweets by keyword/brand, Twitter user, etc. Let me know if you need more details or if that doesn't make sense.