The hard­est chal­lenge when work­ing with dif­fer­ent groups in the opti­miza­tion space is often try­ing to get past their mis­con­cep­tions and to help them view opti­miza­tion in a dif­fer­ent form. It doesn’t mat­ter if they have been doing test­ing for 1 day or 10 years, there is still a mas­sive dif­fer­ence in effi­ciency and the value that can be gen­er­ated. Results are not ran­dom yet so many believe they are because they mis­un­der­stand opti­miza­tion on the most fun­da­men­tal lev­els. The real­ity of real suc­cess­ful opti­miza­tion is often far from the per­ceived real­ity from those just enter­ing the space. The num­ber of mis­con­cep­tions is so large that it can often be nearly impos­si­ble to pri­or­i­tize them or to tackle them all.

Because this prob­lem is so com­mon, I reached out to the smartest peo­ple I know in the indus­try and asked them to share their thoughts about what the one thing they wished peo­ple under­stood about optimization.

Rhett Nor­ton – Consultant

One thing that I wish peo­ple under­stood about suc­cess­ful opti­miza­tion is that test­ing is about dis­ci­pline. To truly be suc­cess­ful you need dis­ci­pline in how to think about test­ing, how to take action, how to orga­nize inter­nally, how to learn iter­a­tively, how to com­mu­ni­cate results, how to learn what influ­ences seg­ments, how to build a pro­gram, and how to cre­ate a cul­ture. It isn’t about launch­ing tests or how many tests you run. It isn’t about cre­at­ing really big tests. It isn’t about per­son­al­iza­tion. It isn’t about mov­ing your polit­i­cal agenda forward.Without dis­ci­pline com­pa­nies go through the motions of test­ing with­out ever really achiev­ing amaz­ing long term results. The most suc­cess­ful com­pa­nies I’ve worked with have been suc­cess­ful with cre­at­ing dis­ci­pline in parts of their test­ing pro­gram. I’ve never seen a com­pany that is dis­ci­plined in every aspect of opti­miza­tion, but hey, maybe your com­pany could be the first.

Drew Phillips – Consultant

I wish that more peo­ple under­stood that opti­miza­tion is a dis­ci­plined, yet free form process. It is dis­ci­plined in that you can’t be suc­cess­ful by sim­ply throw­ing the spaghetti at the wall. Test­ing ran­dom ideas will get you nowhere fast​.It is free form in that you need to have the flex­i­bil­ity to opti­mize ele­ments that you find to be influ­en­tial, not lock your­self into a spe­cific roadmap. Opti­miza­tion is a process that changes as you learn from each cam­paign. You will get the most out of your opti­miza­tion efforts by iter­at­ing off of things you learn from pre­vi­ous tests.

Bran­don Ander­son – Consultant

The one thing I wish opti­miza­tion prac­ti­tion­ers under­stood is the 80/20 rule and the need for focus­ing on the “basics”. 80% of opti­miza­tion ROI comes from doing 20% of opti­miza­tion activ­i­ties. The opti­miza­tion umbrella is get­ting big­ger and big­ger – web, mobile web, mobile app, email, dis­play ad – and the num­ber of activ­i­ties in these areas is almost infi­nite – ban­ners, images, copy, but­tons, lay­out, color, page flow, etc. It’s very easy to get excited about new ini­tia­tives like per­son­al­iza­tion and omnichan­nel. These things may have value. But is their value greater than the “basic” activ­ity of opti­miz­ing page lay­out in the check­out funnel?Sometimes orga­ni­za­tions that have been doing A/B test­ing for years feel like they need to work on com­plex activ­i­ties in order to con­tinue pro­gress­ing. My expe­ri­ence is that even mature orga­ni­za­tions need to look past the hype of new and shiny buzz­words and deter­mine which activ­i­ties will give them the high­est effi­ciency. Get the 80% with 20% of the effort by focus­ing on the basics.

Ryan Roberts – Solu­tion Architect

I wish more peo­ple real­ized that suc­cess­ful opti­miza­tion has to be a process that will require time, effort and thought­ful strat­egy. Just throw­ing together some ran­dom tests misses the point and the ben­e­fit of a well-run opti­miza­tion program.I also wish peo­ple were more care­ful about how they read test results. Peo­ple that rely solely on con­fi­dence cal­cu­la­tions are going to end up with a lot more wrong con­clu­sions than they think. They need to under­stand what the rules of con­clu­sive results should be for their site. And they have to apply them reli­giously to each test they run.

Doug Mum­ford – Consultant

Many great tests don’t (or shouldn’t) take much devel­op­ment time to setup. Orgs should actively work to reduce lead time from idea to launch. Launch­ing a test in under an hour is very pos­si­ble. Orgs tend to anchor their per­cep­tion of devel­op­ment time based on what they’ve done in the past – 4–8 hours for dev slated out two weeks in advance, 3 hours for QA. Why?While there are some tests that will require more time a lot of highly valu­able tests can be done with three lines of CSS or jQuery, loaded up in four browsers to make sure every­thing looks good (and per­haps an iPhone and iPad), and launch. Have a bias for action.

If I had to char­ac­ter­ize my own answer to the ques­tion it would be that there is a mas­sive dif­fer­ence between action and value. Just run­ning a test, be it one or 500, is not the mark that you are suc­cess­fully opti­miz­ing. Opti­miza­tion is about how you tackle large assump­tions, and about how you act on data, and even how you think about what data can and can’t tell you. So much time is wasted in the pur­suit of exe­cut­ing on assump­tions and against the prop­a­ga­tion of agen­das which is the exact oppo­site of where the value of opti­miza­tion comes from.

It is about dis­ci­pline, and sta­tis­tics, and vari­ance, and tech­ni­cal solu­tions, and deal­ing with senior man­age­ment and deal­ing with biases and assump­tions. It is all that and more. It is a means to an ends, but that end is increased rev­enue for your orga­ni­za­tion, not just blindly reach­ing an audi­ence or mak­ing an indi­vid­ual look good. The more you try to jus­tify a spe­cific action or the more com­pli­cated you make some­thing, the less value you get and the more time you waste. Just under­stand­ing that action in and of itself is not the answer is the first step to being truly open to solv­ing the largest chal­lenges that opti­miza­tion pro­grams face. The chal­lenge is never in run­ning tests, the real chal­lenge is find­ing solu­tions and ways to even have these conversations.

What do you find as the one thing you wished peo­ple under­stood about opti­miza­tion? What are you doing to solve it?