One of the fun­nier trick so of the human mind is the want to pigeon hole or describe things in as much detail as pos­si­ble. While there are stereo­types and other harm­ful ver­sions of this, the inverse is usu­ally far more likely to cause havoc with your opti­miza­tion pro­gram, and as such it is the next bias that you need to be aware of; Con­junc­tion Fal­lacy or “the ten­dency to assume that spe­cific con­di­tions are more prob­a­ble than gen­eral ones.”

The clas­sic exam­ple of this fal­lacy is to ask some­one, “which is most likely true about a per­son on your site? Did they come from search, or did they come from your paid search cam­paign code that landed on your #3 land­ing page and who then looked at 3 pages before enter­ing your fun­nel?”. Sta­tis­ti­cally, there is no way for the sec­ond state­ment to be more likely then the first one, since the first one incor­po­rates the sec­ond one and a much larger audi­ence, mean­ing that the scale is mag­ni­tudes greater. Yet we often times find our­selves try­ing to think or do analy­sis in the most detailed terms pos­si­ble, hop­ing that some per­sona or other minute sub seg­ment is some­how more likely to be valu­able then the much larger population.

This men­tal worm tends to make its appear­ance the most often when groups set out to do seg­ment analy­sis or to eval­u­ate user groups. We dive into groups and try to fig­ure out the rate of actions that we want to exploit. Whether it is an engage­ment score, cat­e­gory affin­ity, or even sim­ple cam­paign analy­sis, we dive so deep into the weeds that we will miss a very sim­ple truth. If the group is not large enough, then no mat­ter what work we do, it is never going to be worth the time and effort to exploit it for rev­enue. The other trade for this is the inabil­ity or want to not group these same users into larger groups that may be far more valu­able to inter­act with. Whether it is peo­ple who have looked at 5 cat­e­gory pages and signed-up for newslet­ters or other inef­fi­cient lev­els of detail, you need to always keep an eye on your abil­ity to do some­thing with the data.

This also plays out in your biases towards what type of test you run. Even if inter­net explorer and Fire­fox users may be worth more or more exploitable than cam­paign code 784567 which is only 2% of your users, this bias makes you want to tar­get to that spe­cific group so much more, both as a sign of your great abil­i­ties, but also because we want to be more spe­cific with our inter­ac­tions with peo­ple. Even if the group is much more exploitable, the smaller scale of impact still make it far less valu­able to your site.

Here are some very sim­ple rules for seg­men­ta­tion that will make sure that you com­bat this fallacy:

1) Test all con­tent to all fea­si­ble seg­ments, never pre-dispose that you are tar­get­ing to group X.

2) Mea­sure all seg­men­ta­tion and tar­get­ing against the whole, so that you have the same scale in order to com­pare rel­a­tive impact.

3) All seg­ments needs to be action­able and com­pa­ra­ble, mean­ing the small­est seg­ments gen­er­ally are going to be greater than 7–10% of your pop­u­la­tion depend­ing on your traf­fic volume.

4) Seg­ments need to incor­po­rate more than site behav­iors and direc­tion to the site, try to include seg­ments of all descrip­tions in your analy­sis. Just because you want to tar­get to a spe­cific behav­ior does not mean that behav­iors have more value than non-controllable inter­ac­tions such as the time of day.

5) Be very very excited when you prove your assump­tions wrong on which seg­ment mat­ters most or is the best descrip­tor of exploitable user behavior.

If you fol­low those rules, you are going to get more value from your seg­ment inter­ac­tions and you will stop your­self from falling down this pitrap. We often times have to force a sys­tem on our­selves to insure that we are being bet­ter than we really are, but when it is over, we can look back and see how far we have come and how much we grew because of that dis­ci­pline. Revel in those moments, as they will be the things that give you the great­est value to your­self and your program.

John Hunter
John Hunter

A similar idea (I think) is believing continuing to optimize based on your current users is best. It might be. But you might have wandered into an area where you make 5% of your potential market very happy but those other 95% won't use it. That is potentially a very bad state to be in.

Andrew Anderson
Andrew Anderson

That is not quite what this is about, but that is an example of a form of bounded rationality or of the N-Armed bandit problem. You can never stop trying to grow your market, but there becomes a question of how much do you allocate to growing the market versus how much do you optimize what you have. This really comes down to efficiency, do you make more growing or do you make more optimizing? And how much do I allocate to explore that option on an ongoing basis. Either way, you need to think in terms of what the best way to think about the same users. You can define anyone, anyway you want, with as much or as little details as possible, but when it is all said and done, you have to be able to leverage that definition in the way that generates the most good, as opposed to the way that is creates the most uniqueness.