The hardest challenge when working with different groups in the optimization space is often trying to get past their misconceptions and to help them view optimization in a different form. It doesn’t matter if they have been doing testing for 1 day or 10 years, there is still a massive difference in efficiency and the value that can be generated. Results are not random yet so many believe they are because they misunderstand optimization on the most fundamental levels. The reality of real successful optimization is often far from the perceived reality from those just entering the space. The number of misconceptions is so large that it can often be nearly impossible to prioritize them or to tackle them all.
Because this problem is so common, I reached out to the smartest people I know in the industry and asked them to share their thoughts about what the one thing they wished people understood about optimization.
Rhett Norton – Consultant
One thing that I wish people understood about successful optimization is that testing is about discipline. To truly be successful you need discipline in how to think about testing, how to take action, how to organize internally, how to learn iteratively, how to communicate results, how to learn what influences segments, how to build a program, and how to create a culture. It isn’t about launching tests or how many tests you run. It isn’t about creating really big tests. It isn’t about personalization. It isn’t about moving your political agenda forward.Without discipline companies go through the motions of testing without ever really achieving amazing long term results. The most successful companies I’ve worked with have been successful with creating discipline in parts of their testing program. I’ve never seen a company that is disciplined in every aspect of optimization, but hey, maybe your company could be the first.
Drew Phillips – Consultant
I wish that more people understood that optimization is a disciplined, yet free form process. It is disciplined in that you can’t be successful by simply throwing the spaghetti at the wall. Testing random ideas will get you nowhere fast.It is free form in that you need to have the flexibility to optimize elements that you find to be influential, not lock yourself into a specific roadmap. Optimization is a process that changes as you learn from each campaign. You will get the most out of your optimization efforts by iterating off of things you learn from previous tests.
Brandon Anderson – Consultant
The one thing I wish optimization practitioners understood is the 80/20 rule and the need for focusing on the “basics”. 80% of optimization ROI comes from doing 20% of optimization activities. The optimization umbrella is getting bigger and bigger – web, mobile web, mobile app, email, display ad – and the number of activities in these areas is almost infinite – banners, images, copy, buttons, layout, color, page flow, etc. It’s very easy to get excited about new initiatives like personalization and omnichannel. These things may have value. But is their value greater than the “basic” activity of optimizing page layout in the checkout funnel?Sometimes organizations that have been doing A/B testing for years feel like they need to work on complex activities in order to continue progressing. My experience is that even mature organizations need to look past the hype of new and shiny buzzwords and determine which activities will give them the highest efficiency. Get the 80% with 20% of the effort by focusing on the basics.
Ryan Roberts – Solution Architect
I wish more people realized that successful optimization has to be a process that will require time, effort and thoughtful strategy. Just throwing together some random tests misses the point and the benefit of a well-run optimization program.I also wish people were more careful about how they read test results. People that rely solely on confidence calculations are going to end up with a lot more wrong conclusions than they think. They need to understand what the rules of conclusive results should be for their site. And they have to apply them religiously to each test they run.
Doug Mumford – Consultant
Many great tests don’t (or shouldn’t) take much development time to setup. Orgs should actively work to reduce lead time from idea to launch. Launching a test in under an hour is very possible. Orgs tend to anchor their perception of development time based on what they’ve done in the past – 4-8 hours for dev slated out two weeks in advance, 3 hours for QA. Why?While there are some tests that will require more time a lot of highly valuable tests can be done with three lines of CSS or jQuery, loaded up in four browsers to make sure everything looks good (and perhaps an iPhone and iPad), and launch. Have a bias for action.
If I had to characterize my own answer to the question it would be that there is a massive difference between action and value. Just running a test, be it one or 500, is not the mark that you are successfully optimizing. Optimization is about how you tackle large assumptions, and about how you act on data, and even how you think about what data can and can’t tell you. So much time is wasted in the pursuit of executing on assumptions and against the propagation of agendas which is the exact opposite of where the value of optimization comes from.
It is about discipline, and statistics, and variance, and technical solutions, and dealing with senior management and dealing with biases and assumptions. It is all that and more. It is a means to an ends, but that end is increased revenue for your organization, not just blindly reaching an audience or making an individual look good. The more you try to justify a specific action or the more complicated you make something, the less value you get and the more time you waste. Just understanding that action in and of itself is not the answer is the first step to being truly open to solving the largest challenges that optimization programs face. The challenge is never in running tests, the real challenge is finding solutions and ways to even have these conversations.
What do you find as the one thing you wished people understood about optimization? What are you doing to solve it?