Because of the nature of how most orga­ni­za­tions work, it is com­mon to find test­ing added onto the exist­ing roles or orga­ni­za­tional struc­ture of the ana­lyt­ics group. It makes sense from a very high level view as both dis­ci­plines deal with using data to make deci­sions and both can be viewed as a shared resource through­out the entire orga­ni­za­tion and for each group. The fail­ure comes how­ever when peo­ple who are used to look­ing at and respond­ing to prob­lems like they are ana­lyt­ics issues try to force the same actions onto testing.

The real key to suc­cess in adding opti­miza­tion to your orga­ni­za­tion is in how you tackle the fact that it is a new dis­ci­pline then in the whom or where it fits into that larger pic­ture. Most groups worry about head count and resource and fail to focus on the skills and dif­fer­ent actions that deter­mine suc­cess with opti­miza­tion as opposed to gen­eral ana­lyt­ics or busi­ness intel­li­gence work. The lack of knowl­edge by oth­ers is often abused or lever­aged to help peo­ple gain over­sight onto this project. The real prob­lems lie how­ever when peo­ple do not then adapt usage to meet the new needs and instead sim­ply try and come up with sto­ries as to the value of the program.

Once mature and with key peo­ple in place com­bin­ing the pro­grams can add a lot of value. With­out deep under­stand­ing of those dif­fer­ences how­ever, the inevitable con­clu­sion is less value and wasted time as peo­ple make mis­takes that they do not even know are mis­takes. Resources are a pre­cious com­mod­ity, and the ulti­mate expres­sion of opti­miza­tion is to lever­age them in the most pro­duc­tive way pos­si­ble. In order to ensure this out­come, it is impor­tant that you focus the time of your opti­miza­tion team towards actions that will max­i­mize out­comes and help grow under­stand­ing for your entire organization.

To max­i­mize suc­cess and to ensure that focus is done on the right actions, here is a break­down of the time spent and the value derived from the main actions of a suc­cess­ful opti­miza­tion team.

1) Active Data Acqui­si­tion80% — 90% of the value of most opti­miza­tion pro­grams comes not from the com­monly thought of val­i­da­tion role but in con­tin­ual active data acqui­si­tion and com­par­i­son of fea­si­ble alter­na­tives. It takes time for groups to achieve this role, but when they do the value given to the orga­ni­za­tion increased by mag­ni­tudes. Often this is based around the con­cepts of ban­dit based opti­miza­tion and fragility and is used as an ongo­ing effort to chal­lenge assump­tions and to actively mea­sure the value of dif­fer­ent alternatives.

In this role the opti­miza­tion team con­sis­tently lever­ages low resource efforts that con­sis­tently mea­sure as many dif­fer­ent fea­si­ble alter­na­tives as pos­si­ble and do this across the site. This is in the attempt to max­i­mize the dis­cov­ery of exploitable oppor­tu­ni­ties and the pri­mary role is to chal­lenge assump­tions which oth­er­wise will never be dis­cov­ered. The team needs access to pages and clear rules on mea­sur­ing suc­cess and lever­ag­ing of resources in order to pro­duce con­stant and impact­ful lessons which can shape and direct prod­uct dis­cus­sions and roadmaps.

2) Edu­ca­tion – This is 5–15% of the value derived from the opti­miza­tion group, but pro­vides the abil­ity to do the actions which pro­duce the great­est returns. Because opti­miza­tion requires very dif­fer­ent ways of think­ing about and exe­cut­ing on actions in order to pro­vide the most value this means that one of the key roles for opti­miza­tion is an ongo­ing and con­sis­tent con­ver­sa­tion with groups about dif­fer­ent ways to think about problems.

It is vital that this con­ver­sa­tion always hap­pens prior to any action and is that opti­miza­tion is not just thought of as a sim­ple action in a release cal­en­dar. Groups that fail to think dif­fer­ent are guar­an­teed to get much lower value from their test­ing efforts, waste far more resources, and often have much slower and less pro­duc­tive prod­uct teams over­all. They will get results, they will sim­ply be far smaller results at a much higher cost. Fail­ure to focus on edu­ca­tion often leads group to a purely respon­sive role and leads to pro­grams that are happy with the num­ber of tests they run or by sim­ply pro­duc­ing a sin­gle pos­i­tive result.

There is no such thing as an orga­ni­za­tion that starts out look­ing at the cor­rect things per­fectly and with­out fail someone’s per­sonal agenda leads them to sub­con­sciously search out con­firm­ing actions and data in order to make them­selves look good. Build­ing, main­tain­ing, and edu­cat­ing peo­ple on proper data dis­ci­pline is the sin­gle most con­sis­tent and impor­tant topic of education.

Here are a few of the key top­ics that groups can and should focus on:

i. Rules of Action – Know­ing how to act on data and to be dis­ci­plined in not act­ing too fast or too slow and look­ing at only met­rics that mat­ter to a deci­sion are vital for any data organization.

ii. Sta­tis­ti­cal dis­ci­plines – There are many dif­fer­ent ways to think about data and test­ing and it is vital that peo­ple be exposed and open to dif­fer­ent ways then they are pre­vi­ously aware in order to max­i­mize future growth.

iii. Psy­cho­log­i­cal dis­ci­plines – Opti­miza­tion hits on many psy­cho­log­i­cal dis­ci­plines and con­cepts such as Con­fir­ma­tion bias, Forer effect, Con­gru­ence bias and many others.

iv. Knowl­edge Share – When you are run­ning a suc­cess­ful opti­miza­tion pro­gram you will be con­stantly learn­ing things that go against all pre­vi­ously held beliefs and opin­ions. These lessons learned are the sin­gle most valu­able part of a suc­cess­ful pro­gram and become a core com­po­nent of a pro­gram once it has matured.

3) Ad hoc analy­sis and val­i­da­tion test­ing At most this rep­re­sents 5% of pos­si­ble value pro­vided by test­ing – It is always fun to want to focus on who has the best idea to improve things, but ulti­mately this is the least impor­tant part of a suc­cess­ful pro­gram. The bet­ter the input, the bet­ter the out­put, but only if it is going through a great sys­tem. A poor sys­tem means that it really doesn’t mat­ter how good any idea is.

This is the part that most groups are famil­iar with, where they respond to test ideas directly or ask­ing for more data / details on spe­cific tests.

Gen­er­ally this time is best spent redi­rect­ing towards higher value uses of time and value of data.

Suc­cess­ful pro­grams have their time break­down in the range of:

Active Data Acqui­si­tion / Ongo­ing opti­miza­tion — 60–70% of time
Edu­ca­tion – 15–20% of time
Post hoc analy­sis and val­i­da­tion opti­miza­tion – 15–20% of time

It is gen­er­ally more about the thought and usage of the pro­gram then just who owns it. The real keys to a suc­cess­ful pro­gram are to dif­fer­en­ti­ate the roles and skills. Gen­er­ally if a pro­gram is just start­ing out you they may have 1–2 peo­ple who work on test­ing in some form, with the pri­mary focus on work­ing with dif­fer­ent groups to work with their ideas to pro­vide value. Mature pro­grams might move upwards to 5–10 peo­ple and higher as they con­tinue to grow.

It does no good to add more peo­ple to a prob­lem if you aren’t fix­ing the real prob­lem, which is how the time is spent. More time does not equal more value, bet­ter usage of time means more value. It is never easy to go past your com­fort zone, but that is where you will find all the value. Think about how and where you are spend­ing your time.