Internal search implementation and (a few) best practices

A few weeks ago, a Site­Cat­a­lyst user asked me whether Omni­ture prod­ucts could mea­sure and help opti­mize inter­nal search engine data—the key­words that users are search­ing to find prod­ucts, con­tent, etc. within your site. This was one of those ques­tions that make me smile, because I can answer con­fi­dently and affir­ma­tively. It not only can be done—it prob­a­bly should be done for just about any site fea­tur­ing a search engine. After all, how bet­ter to deter­mine what your users want than by exam­in­ing their search ten­den­cies? There are only a hand­ful of chances in the user expe­ri­ence to learn so much valu­able infor­ma­tion about your user base.

Fig­ure out why inter­nal search terms are valu­able and imple­ment around that

Not sur­pris­ingly, busi­nesses will use inter­nal search infor­ma­tion dif­fer­ently. For exam­ple, a retail site is prob­a­bly inter­ested in the key­words that con­vert most effec­tively into orders, as well as the key­words that return no data (since this can some­times iden­tify holes in your cat­a­log of prod­ucts). A media site, on the other hand, may be more focused on the traf­fic gen­er­ated by each search term, as well as the ban­ner ad click-throughs which fol­low. While the imple­men­ta­tion of inter­nal search mea­sure­ment may be sim­i­lar across dif­fer­ent busi­ness needs, it is nev­er­the­less impor­tant to keep in mind why you care in the first place; it ensures that your imple­men­ta­tion strat­egy will pro­vide the data and key met­rics that you really want.

Use a Cus­tom Con­ver­sion (eVar) vari­able when you want the search term to per­sist, so that sub­se­quent con­ver­sion met­rics can be tied to it. For exam­ple, a user may per­form an inter­nal search at the very begin­ning of his/her visit, but not con­vert until 20 page views later. Use an eVar to allow that search term to receive credit for the order that occurred much later on. Using an eVar will also give you a total count of the num­ber of searches per­formed in the Instances met­ric, allow­ing you to cre­ate a cal­cu­lated met­ric within this report, [Orders] / [Instances], to see which key­words are most and least effec­tive at pro­duc­ing conversion.

A Cus­tom Traf­fic (s.prop) vari­able mea­sur­ing inter­nal search terms, as the name sug­gests, traffic-oriented. It can show you the num­ber of page views, vis­its, and daily/weekly/monthly unique vis­i­tors per key­word on your site. This is ideal for busi­ness that care about searches per visit (or per vis­i­tor), and also those that care about pathing. I’ll dis­cuss pathing by search key­word more below.

It’s worth not­ing that I sus­pect that many of you will want to use both an eVar and an s.prop to cap­ture inter­nal key­words, and this is just fine. In fact, it’s com­mon. Best of both worlds, right?

At least two pop­u­lar imple­men­ta­tion methods

This is described in the Site­Cat­a­lyst Knowl­edge Base:

There are two rec­om­mended approaches for pop­u­lat­ing a Cus­tom Traf­fic (s.prop) vari­able with inter­nal search key­word data. One is to use server-side vari­ables to write out the desired vari­able and search key­word, and the other is to use the get­Query­Param plug-in cap­ture this data out of the query string in the URL of your search results page and pass it into a vari­able. In the exam­ples below, we will use s.prop3 as an exam­ple of a des­ti­na­tion vari­able for your inter­nal search track­ing, but you can use any Cus­tom Traf­fic or Cus­tom Con­ver­sion (s.eVar) vari­able for this purpose.

Server-side approach

The specifics of this method will vary depend­ing on your server-side lan­guage of choice and imple­men­ta­tion. In short, your server should have access the search key­word, either in a GET or a POST vari­able, and you can copy those over to reg­u­lar vari­ables, do any desired manip­u­la­tion, and then write the key­word to the page.

/* You may give each page an identifying name, server, and channel on the next lines. */
s.pageName="Search Results"
s.channel="my site section"
s.prop1="user search"
s.prop2=""
<? echo "s.prop3=\"" . $_GET['keyword'] . "\"" ?>
s.prop4=""
s.prop5=""

If the user had searched for “lit­tle saplings hand­made toys,” the result would be that “lit­tle saplings hand­made toys” would be writ­ten out, and passed into Site­Cat­a­lyst on the page load, as the value of s.prop3:

s.prop3="little saplings handmade toys"

get­Query­Param approach

A more com­mon option is to allow the get­Query­Param plug-in to cap­ture your inter­nal search key­words and to pass them into a vari­able of your choos­ing. The major­ity of site search engine imple­men­ta­tions will give you the user’s key­word in the query string of the results page, and Site­Cat­a­lyst can grab it. For instance, if the user searched your site for “Lit­tle Saplings toys,” the search results page might have a URL sim­i­lar to this:

http://www.yoursite.com/search/results.html?q=little+saplings+handmade+toys

In this case, you could use the get­Query­Param plug-in to search for the value of the “q” para­me­ter and to cap­ture it in s.prop3. (Note that the plus signs are auto­mat­i­cally stripped and replaced with spaces.) For exam­ple, you might include the fol­low­ing within the doPlu­g­ins() func­tion in your Site­Cat­a­lyst code (within the s_doPlugins func­tion in the s_code.js file):

s.prop3=s.getQueryParam('q')

Make sure to stan­dard­ize the case of search terms

The point of pass­ing inter­nal search terms into Site­Cat­a­lyst is to deter­mine the pop­u­lar­ity of var­i­ous val­ues over time and their effect on success—however you define it. As such, you prob­a­bly want to group dif­fer­ent case vari­a­tions of the same term, assum­ing that these vari­a­tions return the same results. You don’t care to see “Lit­tle Sapling hand­made toys” and “lit­tle sapling Hand­made TOYS” as sep­a­rate line items, because the search results (and, thus, the user expe­ri­ence based on this search) is almost cer­tainly going to be the same. (NOTE: For sites with a high traf­fic vol­ume and tons of inter­nal searches, case issues can also increase the total unique val­ues in reports significantly.)

So, if you’re using JavaScript to cap­ture inter­nal search terms, make sure to attach the toLowerCase() to the vari­able that is cap­tur­ing the key­word. For exam­ple, you might do some­thing like this, build­ing on the exam­ple above:

s.prop3=s.getQueryParam('q').toLowerCase();

For server side lan­guages, you would use some­thing like the str­tolower() func­tion in PHP to do the same thing.

Cap­ture the num­ber of search results—especially zero—in a sep­a­rate variable

On top of this, you can pass plenty of other use­ful infor­ma­tion into other con­ver­sion and traf­fic vari­ables. For exam­ple, if your inter­nal search engine returns the count of results for each search, you can cap­ture this infor­ma­tion in an eVar to see how the num­ber of results affects con­ver­sion; are the search tar­geted and accu­rate based on what the user is search­ing for, or do you con­fuse poten­tial cus­tomers with many irrel­e­vant results?

When no results are returned, pass a zero or “null” into this vari­able, so that you can break down “null” by the key­words which returned no results. This will help you under­stand what your users are search­ing for in vain. It can also help under­stand where your prod­uct meta data isn’t speak­ing the same lan­guage as your poten­tial customers.

Use SAINT to com­bine sin­gu­lar and plural key­words (where appropriate)

Just a brief point here: your search engine may return the same results for sin­gu­lar and plural forms of a search term (e.g., the plural “Lit­tle Saplings hand­made toys” ver­sus the sin­gu­lar “Lit­tle Saplings hand­made toy”). In this case, I would rec­om­mend pass­ing the search key­word into Site­Cat­a­lyst as-is, then (if desired) using SAINT clas­si­fi­ca­tions to “group” these sim­i­lar val­ues. The sin­gu­lar and plural forms of the key­word would both be key val­ues in your SAINT upload, and a sin­gle clas­si­fi­ca­tion col­umn, with the same value for both the sin­gu­lar and plural forms of each key­word, would give you an addi­tional report where these vari­a­tions are com­bined into one.

Pathing on search terms

As men­tioned ear­lier, you can have pathing enabled for Cus­tom Traf­fic vari­ables, and this allows you to see how users’ inter­ac­tion with your inter­nal search engine evolves. These reports will dis­play not just indi­vid­ual search key­words and their pop­u­lar­ity, but the actual series of searches per­formed. For exam­ple, what does this “search key­word path” tell you?

There are a few pos­si­bil­i­ties, but these might be users who are inter­ested in pur­chas­ing not only an iMac, but also blue­tooth acces­sories. Do you need to add a “Rec­om­mended Items” sec­tion to help users locate these prod­ucts more easily—so they don’t need to per­form search after search after search? If you already do prod­uct rec­om­men­da­tions, is there a rea­son users aren’t find­ing these acces­sories there?

This is just the tip of the iceberg—internal key­word pathing opens a world of pow­er­ful opti­miza­tion oppor­tu­ni­ties. It is absolutely pos­si­ble, com­pletely cus­tomiz­able to your needs, and fairly straight for­ward for you or your devel­op­ers to imple­ment. As always, please leave a com­ment with any ques­tions, thoughts, or sug­ges­tions that you may have! I’m also avail­able Twit­ter, Friend­Feed, LinkedIn, or by e-mailing omni­ture care [at] omni­ture dot com.