This is the sec­ond post in my series of Five Times When You Should Be Test­ing. The first is when you need to opti­mize beyond the click, you should be doing land­ing page opti­miza­tion. The sec­ond time when you should test, is when you want to resolve an inter­nal dispute.

Resolv­ing inter­nal dis­putes is actu­ally one of the most com­mon rea­sons I see com­pa­nies pur­chas­ing a test­ing solu­tion. Some­body has an idea that the arti­cle detail page on their media web­site would help keep cus­tomers on the site longer and con­sume more ads if it just were re-designed in X or Y way. Some­body else hates the new design idea, and thinks it ought to be kept as-is. Yet a third per­son asserts that both designs are wrong, and she has her own design that she claims is bet­ter than either of the other two.

This debate can go on end­lessly — for years, lit­er­ally. Every­one has an opinion…and they are very will­ing to back up their opin­ion with lots of squishy asser­tions, like “every­one hates Flash,” or “we need an extremely bold call-to-action,” or “we don’t need to say ‘click here’ in our links anymore…that’s so 1999!”

While many of those asser­tions may sound snappy and can really catch hold in your mar­keters’ minds, they aren’t always true. I know a large tech­nol­ogy hard­ware com­pany that had always debated a sim­ple thing like the word­ing of their main calls to action in their hero ban­ners on the home­page. The vis­i­tor would be click­ing through to read the tech spec and pur­chase the item — should we say ‘learn more’ or ‘buy now’? One camp said that ‘buy now’ was too strong for such a high ticket item, and that a softer pitch would attract more poten­tial buy­ers into the top of the pur­chase fun­nel. The other camp said that ‘learn more’ didn’t ade­quately call the vis­i­tor to action, and didn’t exactly make it clear that the vis­i­tor could actu­ally pur­chase the item by click­ing there. Sim­ple solu­tion — run a quick a/b test on your next hero ban­ner pro­mo­tion. Can you guess the win­ner? In this case the ‘learn more’ call to action increased con­ver­sion by almost 50% for those that clicked. I’ve seen it work out the oppo­site way for other sites, so you can’t just take this as a gen­eral rule — you need to test YOUR cus­tomers, not just live off of oth­ers’ find­ings. I ran a sim­i­lar test on my own site, but with ‘more info — buy now’ as a third alter­na­tive. It beat both of the other calls-to-action by almost 30%.

Now, the thing to remem­ber is that when you run tests you are not always going to drive uplift with your new ideas. In fact, fail­ing — and fail­ing fast — can be just as valu­able as find­ing a new design that dri­ves uplift. It saves you from rolling out a poten­tial risk to your exist­ing base of busi­ness. And between rolling out an untested alter­na­tive, and not rolling out any change at all, I think we’d all say that the lat­ter is less risky.

I know one retailer that offered a branded credit card. They knew that if some­one applied and was accepted for their store credit card, that cus­tomer became very loyal and valu­able. Debates had raged inter­nally about whether to offer that credit card much more promi­nently dur­ing the check­out process. There were huge pro­po­nents of the card offer being an inter­sti­tial offer — effec­tively the accep­tance or rejec­tion of this offer was an extra step in the mid­dle of the check­out flow, to ensure that every­one saw the full pitch. Other big retail­ers did this hard-sell approach, and some retail­ers had even attrib­uted their prof­itabil­ity to these credit card pro­grams. When this retailer rolled out the inter­sti­tial offer it did increase credit card signups — but not nearly enough to com­pen­sate for the 5% drop in con­ver­sion, which was sta­tis­ti­cally sig­nif­i­cant for their traf­fic size. They quickly pulled the new check­out flow back and shut off the test. It was a very bad deci­sion averted with min­i­mal adverse impact on revenue.

The tough thing with one of these sit­u­a­tions is that once you’ve sat in a room and brain­stormed cre­ative ideas for how to improve page X or flow Y, your mar­keters start get­ting excited about the new approach. They’re emo­tion­ally invested. They’re get­ting stoked. It’s fresh, and it has all sorts of pos­i­tive argu­ments sup­port­ing it — say­ing why it could be the next great thing. Folks have made com­par­isons to other world-class com­pa­nies that take this same design approach. The design­ers take a stab, and it looks gor­geous. Now they love it, because it is “such a supe­rior visual design” — and let’s face it, many design­ers are Picasso at heart. They love beau­ti­ful design, sleek design. And rev­enue per­for­mance is often a 2nd (or 3rd, or 15th) pri­or­ity. You may even have to involve the devel­op­ers to build new func­tion­al­ity for the new idea, and they invest time & cre­ative juices into it. Every­one is look­ing at this thing and think­ing it really holds poten­tial. And then you roll out the test, and it doesn’t dom­i­nate. That’s a harsh moment. But the thing is, you’ve got to help the team real­ize that fail­ing fast can some­times be just as use­ful as dri­ving uplift. They’ve also got to real­ize that what worked for another site’s vis­i­tors may not be the right thing for your vis­i­tors. You have to test to know what’s right for yours, and that’s a dis­ci­pline that will pay off repeat­edly over time.

Tune in next time for another install­ment of When to Test.

- Brig Graff