For those that read my blogs on a regular basis you will notice that I rarely focus on lists of steps and instead focus on the disciplines and key concepts that dictate making the right choice. The reason for this is that it is easy to get caught up on to-do lists, but knowing what makes a good decision and what makes a bad decision are what will make you successful long term. That being said, it is important from time to time to have a practical list of actions you can take that will dramatically improve your program and that will enable you to get past a lot of the misconceptions and focus on what matters.
Here are 5 actions that you can start doing today that will dramatically shape your program for the better.
1) Choose a single success metric.
This is either the hardest or easiest step for most programs, but it is the single most important thing you can do. Optimizing the wrong metric, while it might make you feel like you accomplished something, does nothing but waste time and effort. If you are optimizing for clicks, bounce rate, dependent metrics (people who clicked on this banner), getting people farther in your site, checking cart entries, or any of a hundred of other misaligned metrics, you are getting little to no value from your program. Never confuse your KPIs from analytics with your testing objective. You cannot assume that just because you get people farther or get more of some action that it magically equals more value for your organization. Stopping all actions until you get this one aligned is fundamental and required for you to provide any value for your organization.
If you are not sure what that metric is, the key is translate action as close to revenue as possible. If you can’t decide, translate every action into a monetary amount, and then only make decisions on RPV. If you are a lead site with equal value per lead, that is the only time conversion rate is acceptable. If you are a media site, score every page (99% of the site scored is the same as 0%) and then use score per visitor. No matter what you choose, the key is that you only look at that metric to make decisions, never get caught reporting on the separate components that produce that value. Never confuse the “goal” of a test with value for the site, only make decisions that help the entire organization, not just your group or your responsibility.
Way too many programs fail or produce fake results because they fail to tackle this problem. That 86% increase in clicks sure sounds nice, but it may be tied to a LOSS in revenue, not a gain, and unless you are trying to lose money, don’t get caught in that trap. A little bit of pain today will save you years of pain during the life of your program. You can get an outcome from your tests without doing this, and often will tell you how great their improvements were, but they have zero idea if those actions actually provided any additional value to the organization as a whole. Just tackling this problem and getting people talking about and aligning on one goal will stop you from wasting time, resources, and allowing others to abuse the program for their political agendas.
If you do nothing else, do this one step.
2) Stop answering every question –
The entire point of aligning on a metric and testing is speed of execution and ensuring that you aren’t getting a false positive result. The longer it takes you to act on something, or the less rational the decision, the lower the value you get. People are always going to want more information; they are always going to want to help their agenda. It is not the testing program’s job to answer every question; it is their job to find the right answer and make sure it is acted on. Never forget that the hardest but most important action you can take is saying no, not yes. Yes makes people happy, no makes people successful.
Agree on rules of action today, and then anytime someone goes off course (and they will), hold everyone accountable and do not give in to their endless questioning. Make sure those rules of action account for all the realities of testing in the real world, not just classroom statistics. If you need to, allow for 3–4 additional metrics, but NEVER make a decision off of those metrics, only use them for future research. Remind people that a single data point neither tells you correlation nor answers why. So much time and effort is wasted on people not understanding what testing can and can’t do. You can get so much more and achieve so much greater results if you just focus on what matters and don’t give in to fear or politics.
There are hundreds of uses of data, almost all of which are futile or pointless. The human mind is wired to use data as selfishly and poorly as possible. Do not let this be the end of the story. Stop reacting to requests and others and change your focus to one of proactive education. Do not get sucked into the context of a current action but instead focus on what they need to know and what poor actions you need to stop.
My suggestion is to start with teaching people about how and why you make a decision (single success metric and rules of action). Follow with what is standard information and how best to use that data (hint, validating things after the fact is not a proper use of information). Other key topics should be correlative versus causal data, cognitive mistakes of data, statistics, and predictive tools of data. You need to make sure these classes cover both what these tools can do, but also just as important how people can abuse these or misunderstand the value of these tools. Don’t confuse teaching people how to get data as the same as how to use data. There are no silver bullets, and tools are only as good as how they are leveraged.
Technology is a lens that either magnifies your strengths or weaknesses, so make sure you are helping others mitigate their weaknesses so they can focus on their strengths. Make it a major focus to be on the offensive on data and make it easier for people to do the right things. It may seem like this is more work, but you will find that doing this will dramatically stop pointless requests and will increase your time available to do what matters.
You are not going to start out knowing everything, nor is not knowing an acceptable excuse. There is always more to learn and new and better ways to think about and tackle problems. Make educating yourself a major priority, and make it something you do every day. How can you expect others to understand things better and use information more rationally if you are not far ahead of them on that quest? Do not focus on just industry blogs, most of them are designed to be cotton candy of the mind, there to be fluff and make you feel better today, but mostly empty and of no value long term. I would actually suggest that if your time is short, read anything but what you would normally look at. Reading something that you are already thinking accomplishes nothing but making you feel better. The goal is to use new outlooks on problems as a way to increase your ability to deal with problems. Look beyond your normal focus and find how others use similar tools for success and failure. There are amazingly gifted people out there, but it takes work to find them and sift through all the fluff that dominates most writing.
The start of my suggested reading lists would be:
Moneyball – Don’t just read this and think, cool, they used data, understand the tools they used, understand what they did and DIDN’T do. Understand efficiencies in market and the closed versus open systems for data analysis.
Black Swan – Any book by Nasim Taleb will do, but there is no greater thinker and explanation of the problems of abusing data and not understanding math can do to the world. This is a bit hard for some to pick up, but I cannot stress enough how important this information is to doing the right thing. To me this is the single most important book for modern data usage and how to design a organization to use it correctly.
You Are Not So Smart – This is my favorite cognitive psychology blog, but any of them will do. Another I might suggest is less wrong. Understand how people abuse data in general, why they believe what they do, and how they will react to data that does not confirm their world view is vital, and you will find the same problems present in the people you work with every day.
MIT Sloan Sports Analytics Conference – As much as Moneyball is a good entry to sports and statistics, it has been passed over many times. You can get a good entry into advance uses of statistics by looking at the work done every year at the Sloan Sports conference.
Don’t stop there. Find other areas, and make sure that you read both what you agree with and what you don’t agree with. Remember that just putting something out there does not make it correct, and that often times what sounds the closest to what you want to hear is designed to do just that, not make you better.
5) Stop Caring about What Wins
There is nothing less important than what won in a test. The winning variant is something you don’t control, but making the right system to discover things is what you do control. Never let yourself get caught up in which variant won, focus instead on making sure you are testing correctly. Do you have the chance to prove yourself and others wrong? Do you understand the math of testing beyond a high school level? Do you know how to act on data? Are you challenging assumptions and focusing on learning over proving people right?
If you do those things, then outcomes will always come. If you don’t, then outcomes are random and they are never as valuable as they should be. Just getting a result from a recipe tells you nothing, it is the ability to build context and act on data that matters. Don’t just think your tool or the goodness of others will magically solve this problem. It is ok if others focus on what wins in a test, as long as that is not the end of the conversation or all that is allowed to happen.
So there you have it, 5 steps that you can start doing this moment to make your program stronger. They aren’t easy, but nothing worth doing ever is. It might seem impossible to add these onto your current workload, but the reality is that doing these frees you up from your current workload and makes each action more valuable. Deal with the real problems, tackle the things no one else wants to, and become better at what you do and you will have the ability to make others better as well.