The art of ask­ing the right questions

Have you ever played the game of Guess Who?  This is the game where you have a crowd of peo­ple and you have to ask your oppo­nent ques­tions to fig­ure out who their name is.  This game, like site opti­miza­tion, is all about ask­ing the right ques­tions.  Con­trast these two players:

Player 1’s Questions

Player 2’s Responses

Does your per­son have base­ball cap with a red stripe?”

Does your per­son have red glass with really pointy tips?”

Does your per­son have a green ear ring?”

”Does our per­son have a blue hat?”





Player 2’s Questions

Player 1’s Responses

Is your per­son Male?”

Does your per­son have a beard”

Does your per­son have a beard from ear to ear?”

Is your per­son David?”





As you prob­a­bly noted, player 1 was ask­ing all the wrong ques­tions.  The art of ask­ing the right ques­tions will not only make you a good Guess Who player, but it will help you be a strate­gic tester.  You can be the best test exe­cuter in the world, but if you ask the wrong ques­tions you will never real­ize the full value of your test­ing program.

The best com­pa­nies with the best test­ing pro­grams don’t worry about test ideas, they focus on test ques­tions.  Ques­tions are impor­tant because they eas­ily trans­late into very clear site learn­ing.  When you test an idea verse another idea, you only know which idea was better—you are still stuck in “bet­ter” test­ing which is the wrong out­come.  In con­trast, when you ask a really good ques­tion it helps you shape the strate­gic setup of the test, opens up the test­ing pos­si­bil­i­ties, allows you to under­stand why some­thing worked, and guides you to where to test next.

As an exam­ple of this I was on a call recently and the client’s senior man­age­ment wanted to test a left nav­i­ga­tion verses a top nav­i­ga­tion with the hope that they could get more links in front of their cus­tomer.  This is idea vs. idea and it is mak­ing a lot of assump­tions.  They are assum­ing that show­ing more links is impor­tant, that the left nav­i­ga­tion is the best place to show links, that the chang­ing the nav­i­ga­tion this way is bet­ter than other nav­i­ga­tion tests we could run, and that the nav­i­ga­tion mat­ters in the first place.  The other prob­lem with try­ing to run a test like this is that in the chance that you do end up with a bet­ter nav­i­ga­tion, you still haven’t learned anything—you don’t really know why it is better.

Decon­struct­ing an Idea

If we were to reframe their idea into a good ques­tion we might go through a process like this. The orig­i­nal idea was to get more links in front of the cus­tomer, but what are the links for?  The links are so that cus­tomers can find what they are look­ing for.  How do peo­ple find what they are look­ing for?  They use lots of things like nav­i­ga­tion, search, inter­nal cam­paigns, fea­tured prod­ucts and cat­a­logs.  After we decon­struct the idea it is easy to see that the ques­tion we should be ask­ing is “How do cus­tomers find prod­ucts they are inter­ested in?”

Once we have the high level ques­tion we would want to find out if the links and nav­i­ga­tion even mat­ter at all. To do this we would ask: How impor­tant is the nav­i­ga­tion rel­a­tive to other shop­ping meth­ods (nav­i­ga­tion, search, inter­nal cam­paigns, fea­tured prod­ucts, prod­uct cat­a­log, etc)? Depend­ing on how this test goes we may learn that the nav­i­ga­tion doesn’t mat­ter at all and we could then test remov­ing it all together.  This helps us use our time and resources to answer the ques­tions that are most valu­able and keeps us from going down rab­bit holes.

Tak­ing the time to think about the assump­tions built into the idea can help refo­cus the ques­tion to one that elim­i­nates assump­tions and gives you the high­est return on your resources invest­ment.  If the nav­i­ga­tion came back as the most influ­en­tial then we would begin to run through the addi­tional ques­tions out­lined below.

· What is the best loca­tion for the nav­i­ga­tion (left, top, right, mul­ti­ple, etc)?

· How should the nav­i­ga­tion func­tion (drop down, fly out, sta­tic, accor­dion, refine­ment, etc)?

· What is the best look and feel of the nav­i­ga­tion (color, shad­ing, text size, icons, etc)?

· What is the best size of the nav­i­ga­tion (big, small, skinny, fat, etc)?

· How many options should we have in the nav­i­ga­tion (Depart­ments, sub­cat­e­gories, sub sub cat­e­gories, etc)?

· What should be in the nav­i­ga­tion that is missing?

· What do we have in the nav­i­ga­tion that should be taken out?

· etc

You have to develop the skill of tak­ing an idea and decon­struct­ing it into some­thing mean­ing­ful that will help you learn.  Often you will get an idea that needs a lot of trans­la­tion to become a valu­able test series.  You have to develop the skill of tak­ing two or three steps back so you can see the big pic­ture.   Like the Guess Who exam­ple, don’t get caught ask­ing ques­tions or test­ing ideas that don’t mat­ter.  There are no per­fect ideas, there are only ques­tions that help you learn about your site.  Back your­self out of the weeds to ask the high­est level ques­tion then work from there.

Cre­at­ing a cul­ture that focuses on ask­ing the right ques­tions so that you learn what works on your site can take time.  One way to do this is to cre­ate a form that any­one can fill out that.  Have them give you their test ideas in the form of a busi­ness ques­tion so that they under­stand that all ideas can be built on.  It is impor­tant to avoid get­ting wor­ried about who is right, and instead have each per­son help inform the sys­tem to make sure that we get the best results and ask the best questions.

Remem­ber that there are a lot of ben­e­fits to ask­ing really good ques­tions.  Good ques­tions help

· You under­stand why some­thing worked (clear learnings)

· Shape the strate­gic setup of the test

· Open up the test­ing possibilities

· Guide you to where to test next

Good luck