The art of asking the right questions

Have you ever played the game of Guess Who?  This is the game where you have a crowd of people and you have to ask your opponent questions to figure out who their name is.  This game, like site optimization, is all about asking the right questions.  Contrast these two players:

Player 1’s Questions

Player 2’s Responses

“Does your person have baseball cap with a red stripe?”

“Does your person have red glass with really pointy tips?”

“Does your person have a green ear ring?”

”Does our person have a blue hat?”





Player 2’s Questions

Player 1’s Responses

“Is your person Male?”

“Does your person have a beard”

“Does your person have a beard from ear to ear?”

“Is your person David?”





As you probably noted, player 1 was asking all the wrong questions.  The art of asking the right questions will not only make you a good Guess Who player, but it will help you be a strategic tester.  You can be the best test executer in the world, but if you ask the wrong questions you will never realize the full value of your testing program.

The best companies with the best testing programs don’t worry about test ideas, they focus on test questions.  Questions are important because they easily translate into very clear site learning.  When you test an idea verse another idea, you only know which idea was better—you are still stuck in “better” testing which is the wrong outcome.  In contrast, when you ask a really good question it helps you shape the strategic setup of the test, opens up the testing possibilities, allows you to understand why something worked, and guides you to where to test next.

As an example of this I was on a call recently and the client’s senior management wanted to test a left navigation verses a top navigation with the hope that they could get more links in front of their customer.  This is idea vs. idea and it is making a lot of assumptions.  They are assuming that showing more links is important, that the left navigation is the best place to show links, that the changing the navigation this way is better than other navigation tests we could run, and that the navigation matters in the first place.  The other problem with trying to run a test like this is that in the chance that you do end up with a better navigation, you still haven’t learned anything—you don’t really know why it is better.

Deconstructing an Idea

If we were to reframe their idea into a good question we might go through a process like this. The original idea was to get more links in front of the customer, but what are the links for?  The links are so that customers can find what they are looking for.  How do people find what they are looking for?  They use lots of things like navigation, search, internal campaigns, featured products and catalogs.  After we deconstruct the idea it is easy to see that the question we should be asking is “How do customers find products they are interested in?”

Once we have the high level question we would want to find out if the links and navigation even matter at all. To do this we would ask: How important is the navigation relative to other shopping methods (navigation, search, internal campaigns, featured products, product catalog, etc)? Depending on how this test goes we may learn that the navigation doesn’t matter at all and we could then test removing it all together.  This helps us use our time and resources to answer the questions that are most valuable and keeps us from going down rabbit holes.

Taking the time to think about the assumptions built into the idea can help refocus the question to one that eliminates assumptions and gives you the highest return on your resources investment.  If the navigation came back as the most influential then we would begin to run through the additional questions outlined below.

· What is the best location for the navigation (left, top, right, multiple, etc)?

· How should the navigation function (drop down, fly out, static, accordion, refinement, etc)?

· What is the best look and feel of the navigation (color, shading, text size, icons, etc)?

· What is the best size of the navigation (big, small, skinny, fat, etc)?

· How many options should we have in the navigation (Departments, subcategories, sub sub categories, etc)?

· What should be in the navigation that is missing?

· What do we have in the navigation that should be taken out?

· etc

You have to develop the skill of taking an idea and deconstructing it into something meaningful that will help you learn.  Often you will get an idea that needs a lot of translation to become a valuable test series.  You have to develop the skill of taking two or three steps back so you can see the big picture.   Like the Guess Who example, don’t get caught asking questions or testing ideas that don’t matter.  There are no perfect ideas, there are only questions that help you learn about your site.  Back yourself out of the weeds to ask the highest level question then work from there.

Creating a culture that focuses on asking the right questions so that you learn what works on your site can take time.  One way to do this is to create a form that anyone can fill out that.  Have them give you their test ideas in the form of a business question so that they understand that all ideas can be built on.  It is important to avoid getting worried about who is right, and instead have each person help inform the system to make sure that we get the best results and ask the best questions.

Remember that there are a lot of benefits to asking really good questions.  Good questions help

· You understand why something worked (clear learnings)

· Shape the strategic setup of the test

· Open up the testing possibilities

· Guide you to where to test next

Good luck