Confidence Level and Its Effects
Confidence level refers a measure of the reliability of a result. A confidence level of 90% (or 0.90) means that there is a probability of at least 90% that the result is reliable. "Significance level" is often interchangeably used.
There is a trade off between how confident you can be in the results of an experiment and how much time and budget it takes to achieve those results. Higher confidence would require more time and budget as it requires more data to be reliable.
By lowering your required confidence level, you allow Upptic's algorithm to more quickly cull poor performing assets, requiring less traffic and less time to reach a decision on the test.
Adjusting the Confidence Level of Tests for an App
By default, when you create an app, Upptic sets the confidence level for its tests to 85%. Based on our ASO experience, we feel that this level achieves the right balance between budget/speed and reliability.
If you're very concerned about budget, you could lower the confidence to 80%. On the other hand, if you're less concerned about budget but want more reliable results, 90% is a good higher value. Note that 95% is getting to super high confidence and should be reserved only for the largest of apps and no budget constraints.
To edit the confidence level for an app, go to the "App Overview" page of that app and scroll to "Details" section:
