For those that read my blogs on a reg­u­lar basis you will notice that I rarely focus on lists of steps and instead focus on the dis­ci­plines and key con­cepts that dic­tate mak­ing the right choice. The rea­son for this is that it is easy to get caught up on to-do lists, but know­ing what makes a good deci­sion and what makes a bad deci­sion are what will make you suc­cess­ful long term. That being said, it is impor­tant from time to time to have a prac­ti­cal list of actions you can take that will dra­mat­i­cally improve your pro­gram and that will enable you to get past a lot of the mis­con­cep­tions and focus on what matters.

Here are 5 actions that you can start doing today that will dra­mat­i­cally shape your pro­gram for the better.

1) Choose a sin­gle suc­cess metric.

This is either the hard­est or eas­i­est step for most pro­grams, but it is the sin­gle most impor­tant thing you can do. Opti­miz­ing the wrong met­ric, while it might make you feel like you accom­plished some­thing, does noth­ing but waste time and effort. If you are opti­miz­ing for clicks, bounce rate, depen­dent met­rics (peo­ple who clicked on this ban­ner), get­ting peo­ple far­ther in your site, check­ing cart entries, or any of a hun­dred of other mis­aligned met­rics, you are get­ting lit­tle to no value from your pro­gram. Never con­fuse your KPIs from ana­lyt­ics with your test­ing objec­tive. You can­not assume that just because you get peo­ple far­ther or get more of some action that it mag­i­cally equals more value for your orga­ni­za­tion. Stop­ping all actions until you get this one aligned is fun­da­men­tal and required for you to pro­vide any value for your organization.

If you are not sure what that met­ric is, the key is trans­late action as close to rev­enue as pos­si­ble. If you can’t decide, trans­late every action into a mon­e­tary amount, and then only make deci­sions on RPV. If you are a lead site with equal value per lead, that is the only time con­ver­sion rate is accept­able. If you are a media site, score every page (99% of the site scored is the same as 0%) and then use score per vis­i­tor. No mat­ter what you choose, the key is that you only look at that met­ric to make deci­sions, never get caught report­ing on the sep­a­rate com­po­nents that pro­duce that value. Never con­fuse the “goal” of a test with value for the site, only make deci­sions that help the entire orga­ni­za­tion, not just your group or your responsibility.

Way too many pro­grams fail or pro­duce fake results because they fail to tackle this prob­lem. That 86% increase in clicks sure sounds nice, but it may be tied to a LOSS in rev­enue, not a gain, and unless you are try­ing to lose money, don’t get caught in that trap. A lit­tle bit of pain today will save you years of pain dur­ing the life of your pro­gram. You can get an out­come from your tests with­out doing this, and often will tell you how great their improve­ments were, but they have zero idea if those actions actu­ally pro­vided any addi­tional value to the orga­ni­za­tion as a whole. Just tack­ling this prob­lem and get­ting peo­ple talk­ing about and align­ing on one goal will stop you from wast­ing time, resources, and allow­ing oth­ers to abuse the pro­gram for their polit­i­cal agendas.

If you do noth­ing else, do this one step.

2) Stop answer­ing every question –

The entire point of align­ing on a met­ric and test­ing is speed of exe­cu­tion and ensur­ing that you aren’t get­ting a false pos­i­tive result. The longer it takes you to act on some­thing, or the less ratio­nal the deci­sion, the lower the value you get. Peo­ple are always going to want more infor­ma­tion; they are always going to want to help their agenda. It is not the test­ing program’s job to answer every ques­tion; it is their job to find the right answer and make sure it is acted on. Never for­get that the hard­est but most impor­tant action you can take is say­ing no, not yes. Yes makes peo­ple happy, no makes peo­ple successful.

Agree on rules of action today, and then any­time some­one goes off course (and they will), hold every­one account­able and do not give in to their end­less ques­tion­ing. Make sure those rules of action account for all the real­i­ties of test­ing in the real world, not just class­room sta­tis­tics. If you need to, allow for 3–4 addi­tional met­rics, but NEVER make a deci­sion off of those met­rics, only use them for future research. Remind peo­ple that a sin­gle data point nei­ther tells you cor­re­la­tion nor answers why. So much time and effort is wasted on peo­ple not under­stand­ing what test­ing can and can’t do. You can get so much more and achieve so much greater results if you just focus on what mat­ters and don’t give in to fear or politics.

3) Edu­cate

There are hun­dreds of uses of data, almost all of which are futile or point­less. The human mind is wired to use data as self­ishly and poorly as pos­si­ble. Do not let this be the end of the story. Stop react­ing to requests and oth­ers and change your focus to one of proac­tive edu­ca­tion. Do not get sucked into the con­text of a cur­rent action but instead focus on what they need to know and what poor actions you need to stop.

My sug­ges­tion is to start with teach­ing peo­ple about how and why you make a deci­sion (sin­gle suc­cess met­ric and rules of action). Fol­low with what is stan­dard infor­ma­tion and how best to use that data (hint, val­i­dat­ing things after the fact is not a proper use of infor­ma­tion). Other key top­ics should be cor­rel­a­tive ver­sus causal data, cog­ni­tive mis­takes of data, sta­tis­tics, and pre­dic­tive tools of data. You need to make sure these classes cover both what these tools can do, but also just as impor­tant how peo­ple can abuse these or mis­un­der­stand the value of these tools. Don’t con­fuse teach­ing peo­ple how to get data as the same as how to use data. There are no sil­ver bul­lets, and tools are only as good as how they are leveraged.

Tech­nol­ogy is a lens that either mag­ni­fies your strengths or weak­nesses, so make sure you are help­ing oth­ers mit­i­gate their weak­nesses so they can focus on their strengths. Make it a major focus to be on the offen­sive on data and make it eas­ier for peo­ple to do the right things. It may seem like this is more work, but you will find that doing this will dra­mat­i­cally stop point­less requests and will increase your time avail­able to do what matters.

4) Read

You are not going to start out know­ing every­thing, nor is not know­ing an accept­able excuse. There is always more to learn and new and bet­ter ways to think about and tackle prob­lems. Make edu­cat­ing your­self a major pri­or­ity, and make it some­thing you do every day. How can you expect oth­ers to under­stand things bet­ter and use infor­ma­tion more ratio­nally if you are not far ahead of them on that quest? Do not focus on just indus­try blogs, most of them are designed to be cot­ton candy of the mind, there to be fluff and make you feel bet­ter today, but mostly empty and of no value long term. I would actu­ally sug­gest that if your time is short, read any­thing but what you would nor­mally look at. Read­ing some­thing that you are already think­ing accom­plishes noth­ing but mak­ing you feel bet­ter. The goal is to use new out­looks on prob­lems as a way to increase your abil­ity to deal with prob­lems. Look beyond your nor­mal focus and find how oth­ers use sim­i­lar tools for suc­cess and fail­ure. There are amaz­ingly gifted peo­ple out there, but it takes work to find them and sift through all the fluff that dom­i­nates most writing.

The start of my sug­gested read­ing lists would be:

Mon­ey­ball – Don’t just read this and think, cool, they used data, under­stand the tools they used, under­stand what they did and DIDN’T do. Under­stand effi­cien­cies in mar­ket and the closed ver­sus open sys­tems for data analysis.

Black Swan – Any book by Nasim Taleb will do, but there is no greater thinker and expla­na­tion of the prob­lems of abus­ing data and not under­stand­ing math can do to the world. This is a bit hard for some to pick up, but I can­not stress enough how impor­tant this infor­ma­tion is to doing the right thing. To me this is the sin­gle most impor­tant book for mod­ern data usage and how to design a orga­ni­za­tion to use it correctly.

You Are Not So Smart – This is my favorite cog­ni­tive psy­chol­ogy blog, but any of them will do. Another I might sug­gest is less wrong. Under­stand how peo­ple abuse data in gen­eral, why they believe what they do, and how they will react to data that does not con­firm their world view is vital, and you will find the same prob­lems present in the peo­ple you work with every day.

MIT Sloan Sports Ana­lyt­ics Con­fer­ence – As much as Mon­ey­ball is a good entry to sports and sta­tis­tics, it has been passed over many times. You can get a good entry into advance uses of sta­tis­tics by look­ing at the work done every year at the Sloan Sports conference.

Don’t stop there. Find other areas, and make sure that you read both what you agree with and what you don’t agree with. Remem­ber that just putting some­thing out there does not make it cor­rect, and that often times what sounds the clos­est to what you want to hear is designed to do just that, not make you better.

5) Stop Car­ing about What Wins

There is noth­ing less impor­tant than what won in a test. The win­ning vari­ant is some­thing you don’t con­trol, but mak­ing the right sys­tem to dis­cover things is what you do con­trol. Never let your­self get caught up in which vari­ant won, focus instead on mak­ing sure you are test­ing cor­rectly. Do you have the chance to prove your­self and oth­ers wrong? Do you under­stand the math of test­ing beyond a high school level? Do you know how to act on data? Are you chal­leng­ing assump­tions and focus­ing on learn­ing over prov­ing peo­ple right?

If you do those things, then out­comes will always come. If you don’t, then out­comes are ran­dom and they are never as valu­able as they should be. Just get­ting a result from a recipe tells you noth­ing, it is the abil­ity to build con­text and act on data that mat­ters. Don’t just think your tool or the good­ness of oth­ers will mag­i­cally solve this prob­lem. It is ok if oth­ers focus on what wins in a test, as long as that is not the end of the con­ver­sa­tion or all that is allowed to happen.

So there you have it, 5 steps that you can start doing this moment to make your pro­gram stronger. They aren’t easy, but noth­ing worth doing ever is. It might seem impos­si­ble to add these onto your cur­rent work­load, but the real­ity is that doing these frees you up from your cur­rent work­load and makes each action more valu­able. Deal with the real prob­lems, tackle the things no one else wants to, and become bet­ter at what you do and you will have the abil­ity to make oth­ers bet­ter as well.

0 comments