Implementing Twitter Data Tracking in Omniture SiteCatalyst

Yes­ter­day, my col­league, Adam Greco, wrote an out­stand­ing post on the solu­tion he debuted last week at Omni­ture Sum­mit, which allows Twit­ter data to be pumped into Site­Cat­a­lyst, Dis­cover, Data Ware­house, etc. using the Omni­ture Data Inser­tion API. Today I’d like to clar­ify the how-to of the solu­tion so that you can get it up and run­ning as smoothly as pos­si­ble to begin report­ing on your brand’s Twit­ter presence.

I won’t pro­vide exam­ples regard­ing the Data Inser­tion API frame­work, because of the wide vari­ety in the types of envi­ron­ments where Twit­ter data can be cap­tured and passed into Site­Cat­a­lyst; the devel­op­ers at your orga­ni­za­tion can learn how to build this API using the resources at devel​oper​.omni​ture​.com. Most impor­tantly, keep in mind that the gen­eral imple­men­ta­tion prin­ci­ples dis­cussed below can be tweaked and altered accord­ing to your unique busi­ness needs, but the fun­da­men­tals regard­ing a.) the cap­ture of this data from Twit­ter and b.) the com­pi­la­tion of this data into a Site­Cat­a­lyst image request are some­what stan­dard across all envi­ron­ments and implementations.

Adam used the exam­ple of a hypo­thet­i­cal web ana­lyst at Com­cast, who was asked by his CMO to pro­vide data sur­round­ing four key busi­ness ques­tions that this solu­tion can answer:

1. How often is your com­pany men­tioned on Twitter?
2. Is there ever a spike (pos­i­tive or neg­a­tive) in brand-related terms (in a week, day or even hourly)?
3. Who are the peo­ple most often men­tion­ing your com­pany on social media tools and who are they com­mu­ni­cat­ing with the most?
4. When are peo­ple on social media tools men­tion­ing key product/service fea­tures that your Prod­uct Man­agers should know about?

Here is how you can obtain answers each of these ques­tions using Twitter’s search API and Omniture’s Data Inser­tion API.

How often your com­pany is men­tioned on tools like Twitter?

Queries to Twitter’s search API (using http://search.twitter.com/search.atom?q=omniture, where “omni­ture” is replaced with a key­word of inter­est to your busi­ness) return XML which fol­lows this gen­eral form:

<entry>
<id>tag:search.twitter.com,2005:1242044829</id>
<published>2009-02-23T19:58:41Z</published>
<link type="text/html" rel="alternate" href="http://twitter.com/jeffjordan/statuses/1242044829"/>
<title>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</title>
<content type="html">RT: Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a></content>
<updated>2009-02-23T19:58:41Z</updated>
<link type="image/png" rel="image" href="http://s3.amazonaws.com/twitter_production/profile_images/70442609/photo_normal.jpg"/>
<twitter:source><a href="http://www.twhirl.org/">twhirl</a></twitter:source>
<author>
<name>jeffjordan (jeffjordan)</name>
<uri>http://twitter.com/jeffjordan</uri>
</author>
</entry>


For each “tweet” (Twit­ter post/comment) returned by this query to the Twit­ter Search API, you would build and post an Omni­ture Data Inser­tion API request sim­i­lar to this one:

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
</request>


where event1 is the event being used to count up the num­ber of Twit­ter brand men­tions. This Data Inser­tion API post, when per­formed for each result returned by the Twit­ter API, would pro­duce a result like the screen shot given in Adam’s post:

A key com­po­nent of this solu­tion is ensur­ing that we do not repeat­edly count his­tor­i­cal searches. If we make a query to the Twit­ter API once every 10 min­utes, we only want to cap­ture the num­ber of searches that occurred dur­ing the last 10 min­utes (i.e. since the last query). This can be done by cap­tur­ing the last num­ber in the <id> ele­ment (fol­low­ing the comma) in the top-most result returned by the Twit­ter API:

<id>tag:search.twitter.com,2005:1242044829</id>

and then putting that num­ber into the “since_id=” para­me­ter in your next Twit­ter API call, as follows:

http://search.twitter.com/search.atom?q=omniture&since_id=1242044829

Per the Twit­ter API doc­u­men­ta­tion, this para­me­ter “returns tweets with sta­tus ids greater than the given id.” In this exam­ple, This ensures that only the rel­e­vant tweets entered since the last API call are returned and entered into your Omni­ture Data Inser­tion API posts.

Is there ever a spike (pos­i­tive or neg­a­tive) in brand-related terms (in a week, day or even hourly)?

To break down the num­ber of tweets, cap­tured in event1 above, by the key­words being tweeted, sim­ply pass the same value used in the q= para­me­ter in your Twit­ter API call into an eVar in your Data Inser­tion API post. Stor­ing the key­word in an eVar also allows you to per­form mul­ti­ple queries of the Twit­ter API to search for mul­ti­ple key­words (e.g. if you have mul­ti­ple prod­ucts), if desired.

In the exam­ple above, the Twit­ter API query is http://search.twitter.com/search.atom?q=omniture. Here, you would sim­ply pass “omni­ture” into an eVar in your Data Inser­tion API post, as fol­lows. (This exam­ple uses eVar5 to store the keyword.)

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
</request>

You would then use Site­Cat­a­lyst Alerts, as Adam sug­gested, to report on trends in brand men­tions on Twitter.

Who are the peo­ple most often men­tion­ing your com­pany on social media tools and who are they com­mu­ni­cat­ing with the most?

As described in Adam’s post, we will use two eVars to store the authors and recip­i­ents of tweets. This infor­ma­tion is con­tained in the XML returned by the Twit­ter Search API, and obtain­ing it is sim­ply a mat­ter of pars­ing cer­tain ele­ments within the <entry> element.

Tweet author data can be obtained using the value of the <name> ele­ment within the <author> ele­ment in the XML exam­ple shown above. For your con­ve­nience, another exam­ple of these ele­ments are repro­duced below.

<author>
<name>OmnitureCare (Ben Gaines)</name>
<uri>http://twitter.com/omniturecare</uri>
</author>

The <name> ele­ment returns both the Twit­ter han­dle (e.g. Omni­ture­Care) as well as the friendly name (e.g. Ben Gaines). If you want to cap­ture just the han­dle (or just the friendly name), you can use a func­tion native to your devel­op­ment envi­ron­ment (such as substr() in PHP) to pull out the por­tion of the string that you need and place it into a vari­able on your server. For consistency’s sake, you may want to prepend “@” to the front of the Twit­ter han­dle cap­tured in this man­ner (e.g. @OmnitureCare).

Tweet recip­i­ent data must be obtained using the <con­tent> tag. Here, you will need to parse the string, look­ing for all sub­strings begin­ning with “@” and con­tin­u­ing until the next space char­ac­ter. For exam­ple, in the Twit­ter XML exam­ple above, the <con­tent> ele­ment has a value of

RT: @Omni_man: Want to learn how to integrate <b>Omniture</b> SiteCatalyst and Twitter? Check out my latest blog post: <a href="http://is.gd/kzLk">http://is.gd/kzLk</a>

The recip­i­ent here is @Omni_man, so you would need to grab this value out of the <con­tent> ele­ment and place it into a vari­able on your server. Adam sug­gested using “No recip­i­ent” as a value when there is no defined recip­i­ent, or (as in PHP) when strpos($content,"@") === FALSE, but you can use any value you want.

Once author and/or receip­i­ent have been cap­tured, you can pass them into Site­Cat­a­lyst in the same Data Inser­tion API post that you are using to count a new men­tion of your brand, as in the exam­ple below. We will use eVar10 to store tweet author and eVar11 to store tweet recipient.

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
<evar10>@JeffJordan</evar10>
<evar11>@Omni_man</evar11>
</request>

This imple­men­ta­tion will allow the kind of report­ing shown in the screen shot below.

When are peo­ple on social media tools men­tion­ing key product/service fea­tures that your Prod­uct Man­agers should know about?

This is actu­ally the most straight­for­ward of the imple­men­ta­tion require­ments given by the CMO in Adam’s exam­ple, because the full text of the Twit­ter post is avail­able in the <con­tent> ele­ment returned by the Twit­ter Search API. Once this text has been stripped of all HTML markup (note the “<” and “>” enti­ties in the <con­tent> ele­ment shown above), all you need to do is put the text into an eVar in your Data Inser­tion API post, as in the exam­ple below (using eVar12 for this purpose):

<?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
<evar10>@JeffJordan</evar10>
<evar11>@Omni_man</evar11>
<evar12>RT: @Omni_man: Want to learn how to integrate Omniture SiteCatalyst and Twitter? Check out my latest blog post: http://is.gd/kzLk</evar12>
</request>

While eVars are lim­ited to 255 char­ac­ters in length, Twit­ter lim­its tweets to 140 char­ac­ters, so they should always be able to be passed into eVars. Hav­ing this full text avail­able in Site­Cat­a­lyst will, as Adam explained, allow you to search for men­tions of a par­tic­u­lar prod­uct, fea­ture, or ser­vice in con­nec­tion with men­tions of your company.

A few other impor­tant tips

• Omni­ture rec­om­mends stor­ing Twit­ter data in its own report suite. The rea­son for this is that the Data Inser­tion API posts described above will count page views, vis­its, and vis­i­tors, but men­tions on Twit­ter are not the same as user inter­ac­tions with your web site. It would be incor­rect (and could wildly inflate traf­fic counts) to inject Twit­ter data into your pro­duc­tion report suite.
• Twit­ter cur­rently does not charge for calls to its search API. Omni­ture does charge for server calls asso­ci­ated with pass­ing Twit­ter data into SiteCatalyst.
• Twitter’s API will return, at most, the 1,500 most recent searches men­tion­ing the keyword(s) that you spec­ify. Keep this in mind as you plan the “sched­ule” accord­ing to which you will be query­ing the API; if your brand or prod­uct is men­tioned very fre­quently, you may need to run queries more fre­quently so that you do not miss results. Use the rpp= para­me­ter to spec­ify how many search results per “page” to return, and the page= para­me­ter to spec­ify what page to start from. These para­me­ters allow you to sift through results when your brand is men­tioned too often to return in a sin­gle query.
• Make sure that the eVars stor­ing the key­word, tweet author, tweet recip­i­ent, and tweet text are fully sub­re­lated accord­ing to the break­downs you will need to per­form. You may not need to fully sub­re­late all four vari­ables. If you want to break down Tweet Author by Tweet Recip­i­ent, but do not need to break down Tweet Recip­i­ent by Tweet Text, then you will need the Tweet Author eVar to be fully sub­re­lated, but the Tweet Text eVar would need only basic sub­re­la­tions. Omni­ture Client­Care can help you set up full sub­re­la­tions on these eVars.
• To obtain an accu­rate unique vis­i­tor (i.e. unique tweet author) count in the report suite stor­ing your Twit­ter data, use a hash on the author han­dle (user­name) for the value in the <vis­i­torid> ele­ment in your Data Inser­tion API posts.
• Using an addi­tional eVar, you can attempt to cap­ture the “mood” of tweets regard­ing your brand by search­ing the <con­tent> ele­ment for the appro­pri­ate emoti­cons, then trans­lat­ing them into a label (e.g. happy) which could be passed into the eVar. If the con­tent of the tweet were “My Omni­ture reports look fan­tas­tic today! ” you could detect the smi­ley face at the end of the tweet (using the strpos() func­tion in PHP, for exam­ple), label this as “happy,” and pass that into an eVar at the time of the Data Inser­tion API post:

  <?xml version=1.0 encoding=UTF-8?>
<request>
<sc_xml_ver>1.0</sc_xml_ver>
<reportsuiteid>yourrsid</reportsuiteid>
<pageurl>http://www.yoursite.com</pageurl>
<pagename>Twitter Mention</pagename>
<events>event1</event>
<evar5>omniture</evar5>
<evar10>@OmnitureCare</evar10>
<evar12>My Omniture reports look fantastic today! </evar12>
<evar13>happy</evar13>
</request>

  Although some sar­cas­tic tweets will likely end up being labeled as “happy,” this can allow you to trend the atti­tudes in per­cep­tion of your brand over time in SiteCatalyst.

Next time, I’ll dis­cuss how you can use the SAINT API to cap­ture Twit­ter data in a slightly dif­fer­ent man­ner; it’s an alter­na­tive method­ol­ogy with some real upside in cer­tain cases.