One of the great frus­tra­tions of an opti­miza­tion group can be work­ing with other groups to build out and opti­mize their cur­rent ini­tia­tives. While the nat­ural out­come of this frus­tra­tion is shy away and just to hand all con­trol of test­ing to each group, result show that this is con­sis­tently the worst way to get value from your opti­miza­tion efforts. To add to this some groups think the only way they can expand is to add resources, instead of improv­ing how and when you do cer­tain actions. In a lot of cases, the most impor­tant bat­tles that you end up wag­ing hav­ing noth­ing to do with get­ting a test live or with shar­ing results, but instead in help­ing grow a new dis­ci­pline through­out an orga­ni­za­tion. So how then do you work within your orga­ni­za­tion to build out the cor­rect dis­ci­plines while mak­ing sure that you are able to test as often and as much as possible?

While there is no sin­gle answer to solv­ing this age­less rid­dle, there are a num­ber of com­mon fac­tors that dif­fer­en­ti­ate the groups that do get real value, from the ones that have to come up with sto­ries to jus­tify their exis­tence. Fun­da­men­tally the abil­ity to grow comes from your abil­ity to have the right con­ver­sa­tions and take the right actions, even when they are not imme­di­ately polit­i­cally pru­dent or make all those that inter­act with the pro­gram com­fort­able. Most pro­grams are halted by the prob­lems present from the issues they do not want to deal with. Deal with those, and you will grease the wheels of expand­ing the pro­gram through­out your organization.

These can often be the hard­est, slow­est, and most polit­i­cal actions you take for your pro­gram, but in the end they are almost always the efforts that pro­duce the great­est returns. The prac­tice of opti­miza­tion is about change, not just in ele­ments in a user expe­ri­ence, but also in the quest to improve how the orga­ni­za­tion itself thinks about and tack­les prob­lems. How these spe­cific dis­ci­plines play out will always take a unique aspect to match your cur­rent orga­ni­za­tion, but the core dis­ci­plines will always remain.

DO – Change the conversation

With­out fail all opti­miza­tion becomes a series of val­i­da­tion actions for var­i­ous groups in a larger orga­ni­za­tion unless you make chang­ing the nat­ural con­ver­sa­tion a pri­or­ity. The prob­lems with allow­ing test­ing to stay here are many, but the largest being the mas­sive inef­fi­cient use of resources, poor and incon­sis­tent results from test, and the inabil­ity to really make a func­tional dif­fer­ence to an orga­ni­za­tion. All groups will try to change opti­miza­tion to meet their cur­rent con­ver­sa­tions, both from famil­iar­ity, and from the abil­ity to own credit for actions they may not have real influ­ence on. They will do this with­out tak­ing equal steps to move their cur­rent con­ver­sa­tions towards opti­miza­tion. Change the con­ver­sa­tion so that both sides are work­ing towards some­thing new and you will find it easy to change actions, fail to change the con­ver­sa­tion, and it will be nearly impossible.

It is vital that peo­ple start to change how they think and how they look at prob­lems if you are going to get any­where close to the value and impact that a suc­cess­ful opti­miza­tion pro­gram can have. Even worse, when cor­rect actions are taken, peo­ple will focus on the fact that their opin­ion led to a change, and mit­i­gate the impact dis­ci­pline and ran­dom­ness really played in find­ing an answer.

Stop talk­ing about win­ners, and instead talk about the rel­a­tive value of all options. Don’t cham­pion the amount of lift from a test, instead cham­pion when assump­tions were proven wrong. Don’t try and tell sto­ries about why some­thing hap­pened, instead talk about what will hap­pen next with results. Stop talk­ing about tests as the end of the con­ver­sa­tion, but sim­ply as a very small part of a much big­ger pic­ture. As a nat­ural end of this change, test ideas will become increas­ingly irrel­e­vant as dis­ci­pline and the cor­rect actions become more important.

More then any­thing, do not let your­self or oth­ers view the action of just get­ting a test live be the defin­ing char­ac­ter­is­tic of your pro­gram and con­ver­sa­tion. These will be dif­fi­cult con­ver­sa­tions at first, as this is not the nat­ural direc­tion that oth­ers would steer the con­ver­sa­tion, but over time if you make it your pri­or­ity, you will find how easy it is to have peo­ple talk about and think about opti­miza­tion in the cor­rect light.

DON’T – Just run an idea some­one brings to you

Decon­struc­tion, the abil­ity to look at the assump­tions that made some­one come to a con­clu­sion, becomes a vital skill in the hands of a great opti­mizer. Every time you tackle an assump­tion, you enable a greater pos­si­ble out­come and also allow each test to pos­si­bly impact far more then intended. Want to test con­tent to a spe­cific user group on your front door? Does con­tent mat­ter most? How about which group? How about where in the user flow? You can eas­ily start build­ing tests to have enough vari­a­tions and to tackle these larger ques­tions, but only if you make it a pri­or­ity. One of the great com­plaints of opti­miza­tion is the con­cept of local max­i­mum, but the real­ity is that this mostly comes from the lim­its of the imag­i­na­tion of the testers, and not from the specifics of the test itself.

DO – Make edu­ca­tion a pri­mary goal of your entire program

From the start, be it sin­gle suc­cess met­ric or just why and how you test, you have to take con­trol and help oth­ers under­stand how and why you are going to tackle prob­lems. To do this, you must talk about dis­ci­pline and how you act out­side of the spe­cific con­ver­sa­tions of a project or test idea, and instead use it as a cat­a­lyst for those con­ver­sa­tions. Stop­ping peo­ple from mea­sur­ing the wrong things can be more impor­tant then what you do mea­sure, just as how you can tackle a test idea is far more impor­tant then the idea itself.

Test­ing seems easy from the out­side, but the real­ity is when you are doing it right it can cause a lot of con­fu­sion and dis­com­fort for a lot of dif­fer­ent mem­bers of each team. You are directly mea­sur­ing that out­come of things and adding account­abil­ity to opin­ions that makes almost every­one uncom­fort­able. You need to be cog­nizant of their wor­ries and cur­rent focus while help­ing them under­stand why you need to do things. Make sure the edu­ca­tion focuses on the value of being proven wrong, so that when it hap­pens people’s egos are not as shat­tered. It is vital these con­ver­sa­tions hap­pen before you take any action, oth­er­wise you will inevitably either sub opti­mize or even worse, get into a polit­i­cal bat­tle between groups.

DON’T – Try to tackle the entire orga­ni­za­tion at one time

There is a famous say­ing, “How do you eat an ele­phant? One bite at a time”. The same holds true for grow­ing your prac­tice within the orga­ni­za­tion. Don’t try to con­vince every­one to do the right thing, instead focus on spe­cific peo­ple or groups, and start with less high pro­file ones. Don’t think facil­i­tate test­ing for the entire orga­ni­za­tion is in any­way suc­cess, as the num­ber of tests you run is not cor­re­lated with the results from a suc­cess­ful pro­gram. No one is going to just change for the sake of change, espe­cially since almost all change you are look­ing to cre­ate is against their own per­sonal empire and agenda. Once you show suc­cess with those smaller groups, and you show how dis­ci­pline played a big role in the out­comes, this will help you con­vince or deal with other groups and even­tu­ally you will find that the entire ele­phant has been eaten.

DO – Make the analy­sis of opti­miza­tion pat­terns far more impor­tant than a sin­gle test

Over time you will start to build out knowl­edge across tests. While it is easy to focus on only what you are deal­ing with in any given day, the real­ity is that the real lessons that you can learn don’t come from a sin­gle data point, but across large num­bers of tests. These lessons, be it what types of changes or where you should focus, become the most valu­able piece as they shape your entire future prod­uct road map. This means that you need to make this infor­ma­tion avail­able, but also cham­pion these larger lessons and help oth­ers under­stand that spe­cific tests prob­a­bly won’t reveal as great of lessons.

DON’T – Report every­thing that every­one wants

This is espe­cially uncom­fort­able for ana­lyt­ics groups that try to add test­ing on as just a func­tion. Part of opti­miza­tion is the dis­ci­pline to focus on what does mat­ter, and to not pre­tend that you can answer the whole slew of ques­tions that come. Peo­ple bought more, but what did they click on? Where did they go? Why did they do what they did? The real­ity is that you have no clue even the cor­re­la­tion from a sin­gle data point, so that data is irrel­e­vant. Peo­ple spent more, but what prod­uct? Again, this will only cause con­flict between spe­cific prod­uct teams and do noth­ing to focus on the change that needs to happen.

The key is to focus on edu­ca­tion and your sin­gle suc­cess met­ric to under­stand why you don’t do these things. It is ok to look at other met­rics, so long as you do not draw ANY con­clu­sions from them in a sin­gle test, and you look at ones that have a chance to pro­vide value (this means not clicks). If you are not the one enforc­ing this dis­ci­pline and express­ing when groups are mak­ing false con­clu­sions, then how can you ever expect oth­ers to do it when it more directly impacts their own agen­das? You will be able to spot pat­terns only after you col­lect a large num­ber of data points and are dis­ci­plined in how you look at the larger pic­ture of your opti­miza­tion efforts.

I am sure that most peo­ple at mul­ti­ple points here asked how real­is­tic these sug­ges­tions are? There are hun­dreds of excuses why peo­ple do things, but the real­ity is that it is up to you to stand up and do the right thing, even if uncom­fort­able, at all times, oth­er­wise you will find it nearly impos­si­ble to do the right thing when it is required. Peo­ple really do want to feel like they are doing the right thing, even if they do not always want to do the right thing. Make it the func­tion of every con­ver­sa­tion on what is the right thing, and then take actions to force dis­ci­pline or make it eas­ier for them to do the right thing, and they will even­tu­ally not know the difference.

If you are proac­tive in your edu­ca­tion, and make these top­ics a focus, you will find it is far eas­ier to push back when and as needed. Remove the bar­ri­ers, find ways to show an impact to the bot­tom line and new ways to tackle prob­lems, and you will find it eas­ier to grow and touch many more parts of the busi­ness. Focus on mak­ing peo­ple happy, just get­ting a test live, not try­ing to grow the con­ver­sa­tion, and you will be sub­ject to all the usual orga­ni­za­tion resis­tance that other efforts face.

0 comments