This article is focussed on explaining:
- Decide how "sensitive" an experiment should be
- Estimate how long an experiment will take
- Prioritize the experiments
Decide how "sensitive" an experiment should be
Once you decide on a hypothesis, you’ll design an experiment. How many variations to test? They are based on your expertise to generate enough data to determine the best choice. The minimum detectable effect (MDE) represents the relative minimum improvement over the original variant.
Minimum detectable effect (MDE, also known as Minimum Detectable Lift) is a number that estimates the minor improvement you’re willing to be able to detect over control. It determines how "sensitive" an experiment is, and in other words, it's an anticipated lift over the control, that can be measured with a degree of certainty.
MDE is the smallest possible change that would be worth investing the time and money to design the test and implement the change permanently on the site. It's important to know: a lower MDEs means increase traffic to ensure that a smaller lift is truly valid.
Use MDE to estimate how long an experiment will take given the following:
- Baseline conversion rate
- Statistical significance
- Traffic allocation
To get the data to evaluate your hypothesis, you need to run an experiment based on a simple calculation based on MDE, visitors and precision.
Usually, the parameters to consider as standard are the following:
- Statistical significance: 95%
- MDE at least 20%
Using these parameters, adding a number of visitors on each variant, benchmark how long to run an experiment and the impact on the business.
Estimate how long an experiment will take
The time for experiments is a crucial factor, and it can be calculated using the following formula, or you can use of the following links:
Prioritize the experiments
Prioritize experiments according to expected ROI; it's a best practice that'll help you generate a fast and easily important impact in revenue and time terms.
You can organize a simple Google Spreadsheet, adding the features related to the prioritization of experiments in the vertical column. For each experiment, you should consider writing the "main KPI factors" to evaluate and sort the ideas through a single score number. For instance, you can consider determining the number of weeks to run it, the time for implementation and the potential business impact to calculate the sorting.
Several frameworks can help you to prioritize the experiments, such as P.I.E. (Potential, Importance and Ease) it will help you to find which tests you should do first, calculating the medium score.