Auto-Allocate : Not showing - Lift Bounds and Confidence

[surjitp43519485]

So I am running a AB test just for few weeks in my development environment.

If I use 50-50 Manual, the Lift Bounds and Confidence Interval for the goal metric and Confidence shows up when I have conversion % for A as 25% and B as 40% .

If I use Auto-allocate, even though A has 64.2% and B has 25% conversion rate, the Lift Bounds and Confidence Interval for the goal metric and Confidence is showing as Blank !!

Any thoughts !!

[dheerajucla]

For any activity, lift bounds start showing up for a given experience when it has reached the state of statistical significance in terms of confidence with respect to control and the confidence interval value.

What differs auto-allocation based activities is that there is no notion of control experience and the statistical significance of each experience is bound to that of the activity itself, which is attained when the true winner of the activity has been determined by the algorithm. Once that happens, lift bounds along with the other statistics - confidence and confidence interval , show up in the report.

Please refer Auto-Allocate documentation for more information around the same.

[surjitp43519485]

Thanks for the clarification.

Just wondering, after running the test for 2 weeks and irrespective of the % of conversion between A & B is it going to show me the Lift Bounds and Confidence ?

[dheerajucla · Accepted solution]

Yes, of course. Once you have the winner , irrespective of how long it takes for the algorithm to determine that depending on the traffic volume and activity, you should start seeing the lift bounds and confidence values.

[surjitp43519485]

Thanks a lot for your prompt response.

What if the test ends before finding any winner.

Ideally my page has pretty less visit so I don't want to run a Manual A/B test and wait for the winner.

The reason I use Auto-allocate is to pick a winner ASAP.

[dheerajucla]

Yes, auto-allocation helps determining the winner faster than a manual AB-test but if the test is stopped before reaching the conclusion i.e. before a winner is declared, one just needs to make the best guess based on the metric values, e.g. conversation rate, by himself. It is neither right, nor very useful, to claim a result when there is still a lot of similarity across the experiences as per the underlying algorithm and the test could go in either direction if run for more time. Hope that helps.