The Road to Greatness: The Do’s and Don’ts of Expanding Optimization Throughout your Organization
One of the great frustrations of an optimization group can be working with other groups to build out and optimize their current initiatives. While the natural outcome of this frustration is shy away and just to hand all control of testing to each group, result show that this is consistently the worst way to get value from your optimization efforts. To add to this some groups think the only way they can expand is to add resources, instead of improving how and when you do certain actions. In a lot of cases, the most important battles that you end up waging having nothing to do with getting a test live or with sharing results, but instead in helping grow a new discipline throughout an organization. So how then do you work within your organization to build out the correct disciplines while making sure that you are able to test as often and as much as possible?
While there is no single answer to solving this ageless riddle, there are a number of common factors that differentiate the groups that do get real value, from the ones that have to come up with stories to justify their existence. Fundamentally the ability to grow comes from your ability to have the right conversations and take the right actions, even when they are not immediately politically prudent or make all those that interact with the program comfortable. Most programs are halted by the problems present from the issues they do not want to deal with. Deal with those, and you will grease the wheels of expanding the program throughout your organization.
These can often be the hardest, slowest, and most political actions you take for your program, but in the end they are almost always the efforts that produce the greatest returns. The practice of optimization is about change, not just in elements in a user experience, but also in the quest to improve how the organization itself thinks about and tackles problems. How these specific disciplines play out will always take a unique aspect to match your current organization, but the core disciplines will always remain.
DO – Change the conversation
Without fail all optimization becomes a series of validation actions for various groups in a larger organization unless you make changing the natural conversation a priority. The problems with allowing testing to stay here are many, but the largest being the massive inefficient use of resources, poor and inconsistent results from test, and the inability to really make a functional difference to an organization. All groups will try to change optimization to meet their current conversations, both from familiarity, and from the ability to own credit for actions they may not have real influence on. They will do this without taking equal steps to move their current conversations towards optimization. Change the conversation so that both sides are working towards something new and you will find it easy to change actions, fail to change the conversation, and it will be nearly impossible.
It is vital that people start to change how they think and how they look at problems if you are going to get anywhere close to the value and impact that a successful optimization program can have. Even worse, when correct actions are taken, people will focus on the fact that their opinion led to a change, and mitigate the impact discipline and randomness really played in finding an answer.
Stop talking about winners, and instead talk about the relative value of all options. Don’t champion the amount of lift from a test, instead champion when assumptions were proven wrong. Don’t try and tell stories about why something happened, instead talk about what will happen next with results. Stop talking about tests as the end of the conversation, but simply as a very small part of a much bigger picture. As a natural end of this change, test ideas will become increasingly irrelevant as discipline and the correct actions become more important.
More then anything, do not let yourself or others view the action of just getting a test live be the defining characteristic of your program and conversation. These will be difficult conversations at first, as this is not the natural direction that others would steer the conversation, but over time if you make it your priority, you will find how easy it is to have people talk about and think about optimization in the correct light.
DON’T – Just run an idea someone brings to you
Deconstruction, the ability to look at the assumptions that made someone come to a conclusion, becomes a vital skill in the hands of a great optimizer. Every time you tackle an assumption, you enable a greater possible outcome and also allow each test to possibly impact far more then intended. Want to test content to a specific user group on your front door? Does content matter most? How about which group? How about where in the user flow? You can easily start building tests to have enough variations and to tackle these larger questions, but only if you make it a priority. One of the great complaints of optimization is the concept of local maximum, but the reality is that this mostly comes from the limits of the imagination of the testers, and not from the specifics of the test itself.
DO – Make education a primary goal of your entire program
From the start, be it single success metric or just why and how you test, you have to take control and help others understand how and why you are going to tackle problems. To do this, you must talk about discipline and how you act outside of the specific conversations of a project or test idea, and instead use it as a catalyst for those conversations. Stopping people from measuring the wrong things can be more important then what you do measure, just as how you can tackle a test idea is far more important then the idea itself.
Testing seems easy from the outside, but the reality is when you are doing it right it can cause a lot of confusion and discomfort for a lot of different members of each team. You are directly measuring that outcome of things and adding accountability to opinions that makes almost everyone uncomfortable. You need to be cognizant of their worries and current focus while helping them understand why you need to do things. Make sure the education focuses on the value of being proven wrong, so that when it happens people’s egos are not as shattered. It is vital these conversations happen before you take any action, otherwise you will inevitably either sub optimize or even worse, get into a political battle between groups.
DON’T – Try to tackle the entire organization at one time
There is a famous saying, “How do you eat an elephant? One bite at a time”. The same holds true for growing your practice within the organization. Don’t try to convince everyone to do the right thing, instead focus on specific people or groups, and start with less high profile ones. Don’t think facilitate testing for the entire organization is in anyway success, as the number of tests you run is not correlated with the results from a successful program. No one is going to just change for the sake of change, especially since almost all change you are looking to create is against their own personal empire and agenda. Once you show success with those smaller groups, and you show how discipline played a big role in the outcomes, this will help you convince or deal with other groups and eventually you will find that the entire elephant has been eaten.
DO – Make the analysis of optimization patterns far more important than a single test
Over time you will start to build out knowledge across tests. While it is easy to focus on only what you are dealing with in any given day, the reality is that the real lessons that you can learn don’t come from a single data point, but across large numbers of tests. These lessons, be it what types of changes or where you should focus, become the most valuable piece as they shape your entire future product road map. This means that you need to make this information available, but also champion these larger lessons and help others understand that specific tests probably won’t reveal as great of lessons.
DON’T – Report everything that everyone wants
This is especially uncomfortable for analytics groups that try to add testing on as just a function. Part of optimization is the discipline to focus on what does matter, and to not pretend that you can answer the whole slew of questions that come. People bought more, but what did they click on? Where did they go? Why did they do what they did? The reality is that you have no clue even the correlation from a single data point, so that data is irrelevant. People spent more, but what product? Again, this will only cause conflict between specific product teams and do nothing to focus on the change that needs to happen.
The key is to focus on education and your single success metric to understand why you don’t do these things. It is ok to look at other metrics, so long as you do not draw ANY conclusions from them in a single test, and you look at ones that have a chance to provide value (this means not clicks). If you are not the one enforcing this discipline and expressing when groups are making false conclusions, then how can you ever expect others to do it when it more directly impacts their own agendas? You will be able to spot patterns only after you collect a large number of data points and are disciplined in how you look at the larger picture of your optimization efforts.
I am sure that most people at multiple points here asked how realistic these suggestions are? There are hundreds of excuses why people do things, but the reality is that it is up to you to stand up and do the right thing, even if uncomfortable, at all times, otherwise you will find it nearly impossible to do the right thing when it is required. People really do want to feel like they are doing the right thing, even if they do not always want to do the right thing. Make it the function of every conversation on what is the right thing, and then take actions to force discipline or make it easier for them to do the right thing, and they will eventually not know the difference.
If you are proactive in your education, and make these topics a focus, you will find it is far easier to push back when and as needed. Remove the barriers, find ways to show an impact to the bottom line and new ways to tackle problems, and you will find it easier to grow and touch many more parts of the business. Focus on making people happy, just getting a test live, not trying to grow the conversation, and you will be subject to all the usual organization resistance that other efforts face.