Let’s start with a summary of what you’ve been reading. In an effort to make “search more secure,” Google is encrypting more search queries through SSL. Searchers signed-in to a Google account are redirected to https://www.google.com (notice the s) and (here’s the rub) search keywords are stripped from the referrer [correction follows] when a natural keyword is clicked. The typed keyword is hidden from you and your site optimization systems (that’s us).

What’s going to change? Natural search keywords and search engines will be under-reported in SiteCatalyst. If you’ve implemented organic search integration in SearchCenter you will see a reduction in organic search term traffic. However, SearchCenter paid keyword data is not affected.

Google engineer Matt Cutts estimates that search encryption will impact less than 10% of organic search queries. To date we have seen that 0.5% to 2% of Google searches are affected, though we expect that number to increase. As a percent of total search traffic, we’ve seen a 0.3% to 1.5% impact.

Rather than hope the 0.5%, 2%, or 10% ratio applies to your site, follow these steps to measure the impact directly:

1. Run the Traffic Sources > Referrers report using the Instance metric

Referrers Report

2. Select the current day in the date selector

3. Filter the report to show referrers from major search engines by adding this to the search box: google.com/search OR google.com/url OR search.yahoo.com OR bing.com. Some tips:

  • Add non-US domain endings, like google.co.uk, where appropriate
  • Don’t forget to capitalize the OR operator
  • In SiteCatalyst 14, this search may take a few minutes

Major Search Engine Filter

4. Note the total in the bottom-right corner of the page (I drop it into Excel, but your memory may be better than mine)

Total

5. Filter the report to encrypted search referrers by searching for google.com/url AND q=& (again, capitalization matters for the AND operator)

Encrypted Search Filter

6. Divide the second total by the first to estimate the change to total search traffic caused by encrypted search.

Because the google.com domain is used for so much more than search, SiteCatalyst only counts a “search” when there is a visible keyword. This prevents mail.google.com from being counted as a search engine.  Unfortunately, this will also prevent Google encrypted searches from being counted as search traffic. As a result, the following SiteCatalyst reports will be impacted:

  • Referrer Types: Search Engine traffic will decrease in favor of traffic for “Other Web Sites”
  • Search Keywords (All, Paid and Natural): Metrics associated with keywords will decrease in favor of “None.”
  • Search Engines (All, Paid and Natural): Metrics associated with Google search engines will decrease in favor of “None.”

We are currently evaluating changes to the processing platform to improve reporting of encrypted search traffic, though there is no way to uncover the original search query.

Impact on Landing Page Optimization

Google search encryption will affect retargeting and landing page adaptation if site content is customized based on the query parameter. However, because search encryption does not apply to all data, there should be an actionable sample of data for both paid and organic search marketers to perform conversion analysis and landing page optimization (especially since Bing and Yahoo do not offer encrypted search options).

For more information, Search Engine Land has a written a detailed post about Google’s recent search encryption changes here: http://searchengineland.com/google-to-begin-encrypting-searches-outbound-clicks-by-default-97435

The esteemed Chris Haleua from the SearchCenter product management team contributed heavily to this post (in other words, if you don’t like it, blame him).

UPDATE (2): Starting Tuesday, 8 Thursday, 10 November 2011, you will see the term “Keyword Unavailable” in the Search Keywords (All and Natural), Marketing Channel Detail and SearchCenter Organic to Paid Keyword reports. Additionally, any loss in Natural Search attribution for Google in the Search Engines report will return to normal levels. This change will not be retroactive. The gradual decrease you’ve noticed in the Search Engines, Marketing Channel, and Referrer Types reports will jump back up by 2-4% 14-20% (according to most customers I’ve spoken with) starting Thursday afternoon, Pacific Time.

UPDATE 3: The changes are live. You will now see “Keyword Unavailable” in keyword reports and Search channel and referrer types will return to normal levels. If you’re using the Channel Manager plugin, you will need to use an updated version.

We are still investigating the status of Unified Sources DB VISTA rules, and hope to have an update on those soon.

Thank you for the comments on this topic, both here and in the Ideas Exchange.

27 comments
Start-up coach
Start-up coach

I savor, cause I discovered exactly what I used to be looking for. You've ended my 4 day lengthy hunt! God Bless you man. Have a nice day. Bye

Joe
Joe

I use the ExcelClient plugin on a weekly (sometimes daily) basis. Up until last month it was working fine with keyword unavailable. I was filtering out "None" (it shows as Unspecified in Excel though) to remove any traffic that was not organic and was getting accurate results. Then last month when I filtered "None" it was removing both Keyword Unavailable and Unspecified which is a drastic reduction in traffic. My question is....what should I enter to filter out the Unspecified traffic? So far I've tried null and Unspecified but neither seems to work. Thanks! Joe

Connell Blackett
Connell Blackett

Great post, however I've realised that by searching for "q=&" we're actually overestimating our encrypted searches, as there are other query string parameters used by Google that match this pattern e.g. "oq=&". Because the search term parameter is always first we're using "?q=&" instead.

Axekap
Axekap

Hi, Wondering whether the percentage of missing keywords vs.organic search is still going up due to Google Plus etc ... or the 15-20 % of total organic is in hte ball park ? Thx !

Jonathan Schwarz
Jonathan Schwarz

Regarding for example this topic, can anybody tell me further message boards or even benchmark circles where companies exchange experiences about web analytics in general and problems like the not provided keyword in particular. Would be great if someone has some information for me. Thanks!

Bret Gundersen
Bret Gundersen

For those of you with the Unified Sources DB VISTA rule in place, we have begun rolling out an update, and will continue to do so over the next few months. Due to the nature of this change, we cannot update all rules at once, so it will take some time before all report suites have been updated. If you need to expedite the process, please work with your account manager to discuss the options available to you.

Derrick
Derrick

Any news on the Unified Sources DB Vista rules? At least a timeline for the fix would be greatly appreciated by all, I'm sure. Thanks!

Kevin Rogers
Kevin Rogers

Has there been anything found out yet about the impact to the Unified Sources VISTA rule?

Franklin
Franklin

Does the 11/10/11 "fix" impact EVAR2 reports?

Scott Calise
Scott Calise

I just want to be sure that the updates you are making will also be made for those who use the Unified Sources DB Vista Rule. We do not use the standard referral reports, but exclusively rely on this Vista rule. Can someone please confirm? Thanks, Scott

Tad Miller
Tad Miller

Seeing 19% of Google Searches missing keyword data as of November 4. So much for "single digit percentages" being effected.

Bret Gundersen
Bret Gundersen

Vedha, There have been no changes to how the filters work in SiteCatalyst. Just make sure you're doing your analysis on the current day's referrers, to avoid running into the unique URL limit Jeff mentioned.

Vedha
Vedha

Did something happen with the way these filters are set up? Until Sunday a client's encrypted searches were between 0.6%-1.7% which spiked to 6% on Monday and 11% on Tuesday. As of 2 Nov, 1:30pm ESP we were still trending at 11%. I want to make sure this methodology still works right before hitting the panic button.

Bret Gundersen
Bret Gundersen

Thanks for the comments, everyone. I'll update this post as soon as we have more information about the change to how we're reporting searches. Jeff, very astute comment about 25k referrer limit for large sites. The good news is that this limit is applied after the day is complete. If you limit your time period to the current day, you'll pull from the full data set.

Scott
Scott

Great post, thanks for being proactive in explaining the implications to us. We use the Unified Sources DB VISTA rule for our referral traffic, does this change also impact that? If so are you also considering a revision to that, similar to the native solution you're exploring for SC? Thanks.

Emily Hill
Emily Hill

@Bret ... weighing in on Hugh's feedback. Yes, identifying it as Encrypted Search bucket would be GREAT. (and even better than Google Analytic's naming of the "Not Provided" bucket, which would surely elicit many questions from users of "What's the difference between 'None' and 'Not Provided' in the Omniture search reports?) I know our SEM team works very hard for every 1-2% of organic search traffic they can grow, so this is a much needed enhancement from that perspective even if we can't see the actual search terms. Hugh logged this on Ideas Exchange, so anyone who agrees - please go Promote it here: http://ideas.omniture.com/t5/Adobe-Idea-Exchange-for-Omniture/Google-logged-in-searches/idi-p/5723

Nate Orshan
Nate Orshan

Cheers Bret & Chris for the explanation. So far for the past week I'm seeing a 2.8% to 3.5% loss of recorded Google organic KW (i.e., "instances of Google searches with a blank 'q' parameter" divided by "total instances of Google searches") across a couple of sites. Not the end of the world...but definitely disturbing, especially given that "Big G" could turn off the referrer spigot completely on a whim. I'd also recommend Danny Sullivan's "Google Puts A Price On Privacy" ( http://searchengineland.com/google-puts-a-price-on-privacy-98029 ) to get a sense of the righteous outrage directed at Google's hypocrisy for starting to cut off organic keyword data to site publishers in the name of protecting user privacy...but not doing likewise for AdWords keyword data.

John Hunter
John Hunter

Very interesting. I am surprised the % of "lost data" will be so low. The article you reference is very interesting. It is good that, at least, google webmaster central will continue to show all keyword data. It is sad to lose nay keyword data though - it is very interesting. Setting to "encrypted search" does seem much better than nothing.

Chris Haleua
Chris Haleua

@Paulsen Thanks for the proactive work on that plugin. A native solution is being considered and we will make sure to update the blog once we know more. @Sylvain The Paid search reports should not be impacted because they should always have a query value. Can you tell us more about how your report is being impacted?

Sylvain
Sylvain

Hi, If Google change only impacts Natural Search, why is it impacting Omniture Paid Search Keywords and Engines report? Thank you, Sylvain

Bret Gundersen
Bret Gundersen

Hugh, you're right. There is enough information in the referrer to identify this as natural search and we are currently evaluating a SiteCatalyst update. We could, for example, just set the keyword value to "Encrypted Search" when it's empty. I'm interested in your thoughts on what would make the most sense to users.

Hugh W
Hugh W

Is there going to be an update to sitecatalyst so that visits satisfying the conditions in point 5 can be correctly attributed in the search engines and referrer types report? The lack of a keyword shouldn't impact the ability to identify these visits as from Google organic search as there is still sufficient information in the referrer to differentiate these from google non-search (such as iGoogle).

Jonathan Kay
Jonathan Kay

Very useful post - thanks. Happy to share that I'm seeing a 2.5% impact for one of my UK clients when looking at google.com traffic to their .co.uk site (for this morning). Google hasn't yet made the change to google.co.uk

Bret Gundersen
Bret Gundersen

Scott, we are still evaluating whether and how to update existing DB VISTA rules, but I will update this post when I have more information.