Becoming Spider-Man:Testing Search Engine Traffic [Inside Adobe SiteCatalyst]
One of the cool out-of-the-box features of Adobe’s SiteCatalyst is the ability to track how visitors are finding your site. There are numerous methods by which visitors might come to your site. Some know your site’s domain or have your site bookmarked. Some come to your site via links from other sites. One of the most coveted sources for traffic comes from search engines, including paid search and natural (or organic) search.
SiteCatalyst uses information from existing image requests on your pages, does some processing of the data collected, and then displays the information in the Traffic Sources reports. This means that there is no additional effort you need to undertake to collect this data. Kinda feels like turning the tables on those search spiders doesn’t it? Besides, who doesn’t dream of being our favorite web-slinging hero (especially my 3-year old son).
SiteCatalyst is able to capture data from this traffic using three key data points.
1) The referrer (reported by the browser in the HTTP response header) is the URL from the page just prior to the current page. The referrer is used to populate the “Referrers,” “Referring Domain,” and “Referrer Type” reports. We can detect which search engine a visitor came from by the domain in the referrer (yahoo.com, google.com, etc.).
2) The referrer also contains the keywords used in the search performed. This is contained in the query string of the referrer. The parameter used varies by search engine. Distinction between paid keywords and natural keywords can be accomplished using paid search detection (see #3).
3) In the case of paid search, a campaign tracking code is included in the query string of the destination URL. This is used not only to determine the marketing campaign, but also to distinguish between traffic from paid search campaigns (SEM efforts) and traffic from natural search (SEO efforts). Configuration of paid search detection is handled in the Admin Console of a particular report suite.
Typical website development cycles include three different environments: a development environment for the initial website build, a staging environment for validating and quality assurance, and a production environment for the actual end users. The data from your end users is the most valuable to gain actionable insights for optimizing your site. As such, Adobe recommends separating data collected from production environments from that in development/staging environments.
This presents a bit of a problem in regards to testing search engine traffic. This traffic source can be very difficult to test in a development environment because the dev site will typically be on an internal network and/or not submitted to search engines for indexing. Fortunately for you, there are a few workarounds you can use for testing this functionality.
1) Change the domain resolution via the hosts file on your computer.
2) Use a DOM Manipulation Tool to adjust the destination URL of a search result.
I’ll outline both approaches briefly below.
1) Navigate to the hosts file on your computer – the exact location for your operating system can be found at Wikipedia (http://en.wikipedia.org/wiki/Hosts_(file)#Location_in_the_file_system)
2) Insert a value to point your production domain name to your testing IP address
a. If you don’t know the ip address of your testing server, you can find it by opening a command or terminal window (Windows-r then type “cmd” in Windows). Type “ping [test.mysite.com]” replacing the domain name with your testing domain. The 4 numbers separated by periods constitute the IP address. You want to ensure that the test IP address maps to the production domain name.
3) Save the updated version of the hosts file.
4) Try clicking through on a search result from your favorite search engine.
1) Install Firebug for Firefox or some other DOM Manipulation Tool.
2) Search for common keywords on your favorite Search Engine that will lead you to your website.
3) Right Click on the link to your site. Click on “Inspect Element.”
4) Click on the destination URL and adjust the value to your testing environment (i.e. http://test.mysite.com).
5) Click on the link in the web pane.
6) Use the DigitalPulse Debugger to verify that the referrer contains the search information (domain and keywords).
The ability to analyze search engine traffic is one of the more valuable functions of the Adobe Online Marketing Suite. It is essential to ensure your SiteCatalyst implementation is configured correctly to capture this valuable data.
Have specific questions about Adobe SiteCatalyst? Want to track a data point on your website, but not sure where to start with the implementation? Follow me on Twitter @sitecattips Please feel free to leave a comment here or send me an email at adobesitecatalyst (at) adobe.com