You are dri­ving down a road when your GPS tells you to turn left. You make a sud­den motion, find­ing your­self down a small side road. It doesn’t look like where you are try­ing to go, but you have to fol­low your GPS; oth­er­wise, you will get lost. You con­tinue, then your GPS tells you to go right. There isn’t a road there, and because you are stuck doing only what the GPS tells you, you turn and sud­denly find your­self run­ning off a cliff, fly­ing to your demise in a canyon below. Sound like a bad TV skit? The real­ity is that this is how most peo­ple lever­age their “roadmaps” in terms of how they run their opti­miza­tion programs.

While hypoth­e­sis is still the most mis­un­der­stood term in all of opti­miza­tion, the most abused may be roadmap. So many dif­fer­ent groups claim they have a roadmap or to be fol­low­ing a roadmap or that it is on their “roadmap” and yet so few under­stand how one is meant to be used. A roadmap (lit­tle r) is a list of tests, most of which serve as a great excuse to waste time and effort and to get locked into a sys­tem of projects. A Roadmap (cap­i­tal R) is a con­stantly shift­ing list of pri­or­i­ties by which you will cre­ate actions and act to dis­cover where to go next. This dis­tinc­tion is fun­da­men­tal if you have any hope of really achiev­ing great results with your pro­gram, and yet so many hap­pily focus on the first for the sake of inter­nal processes or the inabil­ity to change how their opti­miza­tion pro­gram oper­ates in pro­duc­ing revenue.

Let’s start with what the goal of opti­miza­tion is. It is not to run a test.

Tests are a means to an end.

The goal of an opti­miza­tion pro­gram is to pro­duce the max­i­mum amount of rev­enue for what­ever resources you spend on it. The same is true of every effort you do, be it per­son­al­iza­tion, SEO, con­tent cre­ation or a pro­mo­tion. You are not just doing it because it is fun, you are doing those things to increase the rev­enue to your orga­ni­za­tion. This means that those are just tac­tics and not the end onto itself. This is fun­da­men­tal to under­stand­ing the dif­fer­ence between a roadmap and a Roadmap.

Any­time we con­fuse the action for the end goal, we lose almost all pos­si­ble value because we have lost the abil­ity to go in any other direc­tion. When we get stuck on a review process and a large series of tests you are mak­ing the deci­sion to focus on the action and not the value it gen­er­ates. You become a means to empty action, not a means to the end of gen­er­at­ing rev­enue. You are say­ing, at that point, that you couldn’t care less if you make money, so long as these few spe­cific tests get run.

If you instead focus on the end goal, then the first and most impor­tant piece is to dis­cover how best to do that. You may have some test ideas and some things you are going to exe­cute on, but they are fun­gi­ble. You must and will con­stantly shift them as you learn more and as you go in new direc­tions. You can­not be stuck on the path if the end goal is the most impor­tant, you must instead focus on the dis­ci­pline and flex­i­bil­ity to go any­where the data tells you.

This is why a Roadmap is just a series of places to focus. It might be on per­son­al­iz­ing an expe­ri­ence, or improv­ing a prod­uct page, or on improv­ing your rec­om­men­da­tion sys­tem, but that is what you are try­ing to do. You are hop­ing that doing that will result in more rev­enue, but you are not tied to spe­cific tac­tics, just find­ing the best way to accom­plish the end goal. Often times you will have no more then 1 or at most 2 tests for each area when you start, but you plan out the time to shift and the time to con­tinue down any path that presents itself to you. From there you can work out actions which will pro­duce answers, things like inclusion/exclusion test­ing, or MVT, or con­tent serv­ing so that you can mea­sure the value of dif­fer­ent alter­na­tives. At that point, you then focus on what­ever the answers you have are and con­tinue to drive for­ward based on those results.

The amaz­ing or frus­trat­ing part of this, depend­ing on which approach you are used to, is that you never know where you will end up. You might end up with a dynamic lay­out for your prod­uct page, or tar­get­ing con­tent based on time of day, or on remov­ing your rec­om­men­da­tions sys­tem from a page. The far­ther you end up from where you imag­ined the more rev­enue you make. Each step that takes you in a new direc­tion can only do so by prov­ing using ratio­nal mea­sure­ments that it out­per­forms where you thought you were going to go. You can end up just about any­where and that is what makes it so powerful.

The most com­mon refrain you get when tack­ling prob­lems this way is that it is hard to plan resources, but that argu­ment just does not hold water. You know you are going to test and you know you are going to need resources. This just means you plan time. What you aren’t plan­ning on is that time being spend on cod­ing this one spe­cific mod­ule 6 months from now. The action of that time is con­stantly shift­ing and updat­ing, it isn’t set in stone. you can plan resources extremely eas­ily. What you can’t do how­ever is focus those resources only on one per­sons opin­ion or on a sin­gu­lar person’s agenda. It is not that you spend more resources or can’t plan, you just spend them dif­fer­ently and away from empty talks about a test and about build­ing a suc­cess­ful and mean­ing­ful program.

The real chal­lenge becomes not resource plan­ning but account­abil­ity. So many pro­grams hold onto their list of tests because it jus­ti­fies their actions. It becomes about check­ing off that a test was done and not about the effi­ciency or the value of that test. At the end of the day the peo­ple in your pro­gram get to choose between their own account­abil­ity between just run­ning tests or with actu­ally pro­vid­ing value. If you are focus­ing on an empty series of tests, then you will always just be doing action. If you can instead view your Roadmap as a con­stantly shift­ing series of actions that focus only on the value they derive, then you will never worry about any spe­cific test or about try­ing to val­i­date test ideas.

In real­ity the biggest chal­lenge to tack­ling prob­lems like this is the ego of the peo­ple in your pro­gram and the exec­u­tives who might be involved. Peo­ple pro­tect them­selves at all cases because account­abil­ity is the scari­est thing in the world for most peo­ple. The old sys­tems have every­thing going through them and with their bless­ing is every­thing done. When you are going wher­ever the data takes you then you are faced with going in a direc­tion that might not be where that exec­u­tive thought of 3 weeks ago. When you just focus on your part of the a lager process or when you accept their divined vision as the only means to an end then you have essen­tially said that you have no value at all to the orga­ni­za­tion and are just a fun­gi­ble means to an empty end.

This is why an edu­ca­tion pro­gram and why a focus on dis­cover is so vital for the value derived from your test­ing pro­gram. Man­age­ment might view this as a loss of power but the real­ity is that it is so much more. They aren’t con­strained by some ran­dom thought they had, no mat­ter how great it was, and can instead encour­age oth­ers to expand on their cre­ativ­ity. It is no longer about hav­ing the right answer but about mea­sur­ing and going with the best ideas your entire team can come up with. You can tell just how far you are from this point with the num­ber of empty I believe/I think/I feel con­ver­sa­tions you hear in meet­ings. The less you hear of those the closer you are to achiev­ing real value. It isn’t about a review process but instead about the cre­ation process and the man­age­ment of the sys­tem to ensure ratio­nal deci­sion making.

So many orga­ni­za­tions are led to drive into that canyon or into a ran­dom lake. Even worse there are always peo­ple at those orga­ni­za­tions who will describe that water they are drown­ing in as the expected des­ti­na­tion. If you really want to go to new places and really want to end up where you should then you are going to need to give up your belief in that roadmap that you hold so dearly to. Find your own Roadmap, let it shift and go where it needs to, and you will be amazed as just how far you can go and how many new sights you will see.


Hi all, And i always thought marketing was something boring. But it is really exiting and it is so big. You may understand some things about Internet marketing. It first collects data or information about the marketing problem, then it records this data, analyzes this data, then it draws conclusions about this data, later it gives suggestions or advice for solving the marketing problem. Just sit and think about how you would tell a friend about something, then put that down in text.




Senior Marketing Manager @ Fourquadrant