Reducing the Costs of your Product Experiments
How to Stop Wasting Money on Futile AB-Tests
Ever run an AB-Test without any idea what to do with the results after the test finished? Or maybe you already knew what you wanted to do (or had to do) before the experiment even started?
Running experiments that don’t lead to a business decision is a waste of money and time. But how can we decide which ideas are worth the investment to launch them as an experiment to our users?
In this article, I’ll explain
- how to decide whether an idea should be crafted into an experiment
- how tests we want to run can be optimised to reduce costs
Why do we have to choose?
Experiments require different types of investments. Although I am a big advocate of testing and experimenting with new ideas, it is vital to keep these costs in mind. In my experience, the most substantial cost factors in the test execution phase are implementation costs, revenue and time:
1) Implementation Costs
Probably the most visible cost factor are implementation costs. AB-Tests have to be developed by a development team that wants to be paid. Those costs can be neglected if you’re only testing small UI changes that can be implemented using a visual editor provided by most AB-Testing tools.
2) Revenue
Showing an inferior variant to a significant portion of your website’s traffic can have a direct impact on your business figures. For example, a drop in conversion rate by a few per cent in your test group can translate into a severe decline in revenue. Vice versa, there are also opportunity costs when we’re not showing a better performing variant to all of our users for the time the test runs.
3) Time
Experiments take time to deliver significant results to inform decisions. This can slow down the decision-making process and reduce the number of improvements you implement and release per year.
Some hypotheses might be conflicting or exclude each other so that users can only be exposed to one experiment. Running such experiments in parallel means…
