“We Don’t Run AB-Tests Any more.”
Why People Abandon the Most Important Tool in Product Development
AB-Testing can be frustrating. You put your allegedly revolutionary idea out in the wild, only to see that it doesn’t make any difference. And maybe you had the same experience with the last five experiments you ran.
“We don’t run AB-Tests any more, because of all the experiments we ran, only a couple delivered significant results.”
It is easy to blame the mechanism behind it and refrain from AB-Testing. In the end, isn’t AB-Testing just an additional amount of work to test something we want to launch anyway? And a small design update isn’t expected to achieve any significant results anyway, so no reason to put in the extra effort.
“Let’s launch this change without testing it. The change is too small to show a significant impact.”
This rationale couldn’t be more wrong.
First of all, AB-Testing is not about achieving significant results.
Experimentation is a mechanism to manage business risk and quantify the impact of a new initiative or change. It makes sure we launch the best…
