One of my favorite online blogs is You are Not So Smart. I find it fascinating the fallacies that make up our day to day lives, and find that the number one driver of changing client cultures is to tackle and teach lessons that help them challenge or break some of these fallacies.
The first of these biases is CONGRUENCE BIAS, or “the tendency to test hypotheses exclusively through direct testing, in contrast to tests of possible alternative hypotheses”, or otherwise known as, trying to prove myself right. You see this all the time in testing groups, as they test only 1 or 2 things that are very similar, then get a small win and claim it is the greatest thing since sliced bread. You also see this when they know they are going to test targeting to return users, or changing the product image on their landing pages, or any predetermined priority of elements or changes. We are all guilty of this, and it takes discipline and dedication to not let it run your program or your life.
There are entire sub industries who feed and live off of this fallacy, like sites that show winning tests, many “top” blogs, or people or agencies who sell “we have all the answers”. Even internally, how often have you sat around having “I feel/I believe” conversations to determine what the best course of action is (and as a side, why do people think those two statements mean something different in that context?). You also see the same thing when you have groups who all they want to do is talk about tests, not about their program. It means that they are so caught up on trying to prove themselves right that they are missing the forest for the leaves.
It is fascinating how often the groups that get some of the worst ROI for their testing are the ones who champion and talk about it the most, and the reason for it is this bias. The people in charge are trying to make themselves look good (I had the right answer) instead of finding the best answer. The answer is simple… know you are wrong… even if you have the perfect “best” answer today, you are going to be wrong in a number of months. If you don’t have the context for the result you have, or if you don’t hear the context for the result that someone is pushing on you, then you do not know anything about the value of the outcome.
If you want to make sure that you are both getting the most value, and that you are letting a program learn and grow to the best conclusion, the choice is simple, challenge this bias. Include the null assumption, include every feasible alternative, even if you think it won’t win (there is an almost inverse correlation between what people think will win and what will win). Challenge that just because one person out there got a 10% lift doing something, that it is the best thing for you? How do you know how many other people failed miserably doing it? How do you know that even if they got the 10% that they couldn’t have gotten 15 or 20% by doing something simpler and with less effort?
Challenge yourself… Challenge how you think and challenge how others think and you will find the next “truth”. Let the reality of the data (the CASUAL data) tell you where to go and what to do, don’t let hubris or popular opinion tell you the value of something.
The system is what gets you values anyways, not any individual an action.
Why we do what we do: Garbage in and Garbage Out — Congruence Bias
One of my favorite online blogs is You are Not So Smart. I find it fascinating the fallacies that make up our day to day lives, and find that the number one driver of changing client cultures is to tackle and teach lessons that help them challenge or break some of these fallacies.
The first of these biases is CONGRUENCE BIAS, or “the tendency to test hypotheses exclusively through direct testing, in contrast to tests of possible alternative hypotheses”, or otherwise known as, trying to prove myself right. You see this all the time in testing groups, as they test only 1 or 2 things that are very similar, then get a small win and claim it is the greatest thing since sliced bread. You also see this when they know they are going to test targeting to return users, or changing the product image on their landing pages, or any predetermined priority of elements or changes. We are all guilty of this, and it takes discipline and dedication to not let it run your program or your life.
There are entire sub industries who feed and live off of this fallacy, like sites that show winning tests, many “top” blogs, or people or agencies who sell “we have all the answers”. Even internally, how often have you sat around having “I feel/I believe” conversations to determine what the best course of action is (and as a side, why do people think those two statements mean something different in that context?). You also see the same thing when you have groups who all they want to do is talk about tests, not about their program. It means that they are so caught up on trying to prove themselves right that they are missing the forest for the leaves.
It is fascinating how often the groups that get some of the worst ROI for their testing are the ones who champion and talk about it the most, and the reason for it is this bias. The people in charge are trying to make themselves look good (I had the right answer) instead of finding the best answer. The answer is simple… know you are wrong… even if you have the perfect “best” answer today, you are going to be wrong in a number of months. If you don’t have the context for the result you have, or if you don’t hear the context for the result that someone is pushing on you, then you do not know anything about the value of the outcome.
If you want to make sure that you are both getting the most value, and that you are letting a program learn and grow to the best conclusion, the choice is simple, challenge this bias. Include the null assumption, include every feasible alternative, even if you think it won’t win (there is an almost inverse correlation between what people think will win and what will win). Challenge that just because one person out there got a 10% lift doing something, that it is the best thing for you? How do you know how many other people failed miserably doing it? How do you know that even if they got the 10% that they couldn’t have gotten 15 or 20% by doing something simpler and with less effort?
Challenge yourself… Challenge how you think and challenge how others think and you will find the next “truth”. Let the reality of the data (the CASUAL data) tell you where to go and what to do, don’t let hubris or popular opinion tell you the value of something.
The system is what gets you values anyways, not any individual an action.