Auto-Allocate : Not showing - Lift Bounds and Confidence | Community
Skip to main content
surjitp43519485
surjitp43519485
Level 2
February 8, 2018
Solved

Auto-Allocate : Not showing - Lift Bounds and Confidence

  • February 8, 2018
  • 5 replies
  • 3521 views

So I am running a AB test just for few weeks in my development environment.

If I use 50-50 Manual, the Lift Bounds and Confidence Interval for the goal metric and Confidence shows up when I have conversion % for A as 25% and B as 40% .

If I use Auto-allocate, even though A has 64.2% and B has 25% conversion rate, the Lift Bounds and Confidence Interval for the goal metric and Confidence is showing as Blank !!

Any thoughts !!

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by dheerajucla

    Yes, of course. Once you have the winner , irrespective of how long it takes for the algorithm to determine that depending on the traffic volume and activity, you should start seeing the lift bounds and confidence values.

    5 replies

    D
    dheerajucla
    Level 2
    February 9, 2018

    For any activity, lift bounds start showing up for a given experience when it has reached the state of statistical significance in terms of confidence with respect to control and the confidence interval value.

    What differs auto-allocation based activities is that there is no notion of control experience and the statistical significance of each experience is bound to that of the activity itself, which is attained when the true winner of the activity has been determined by the algorithm. Once that happens, lift bounds along with the other statistics - confidence and confidence interval , show up in the report.

    Please refer Auto-Allocate documentation for more information around the same.

      surjitp43519485
      surjitp43519485Author
      Level 2
      February 9, 2018

      Thanks for the clarification.

      Just wondering, after running the test for 2 weeks and irrespective of the % of conversion between A & B is it going to show me the Lift Bounds and Confidence ?

        D
        dheerajuclaAccepted solution
        Level 2
        February 9, 2018

        Yes, of course. Once you have the winner , irrespective of how long it takes for the algorithm to determine that depending on the traffic volume and activity, you should start seeing the lift bounds and confidence values.

          surjitp43519485
          surjitp43519485Author
          Level 2
          February 9, 2018

          Thanks a lot for your prompt response.

          What if the test ends before finding any winner.

          Ideally my page has pretty less visit so I don't want to run a Manual A/B test and wait for the winner.

          The reason I use Auto-allocate is to pick a winner ASAP.

            D
            dheerajucla
            Level 2
            February 13, 2018

            Yes, auto-allocation helps determining the winner faster than a manual AB-test but if the test is stopped before reaching the conclusion i.e. before a winner is declared, one just needs to make the best guess based on the metric values, e.g. conversation rate, by himself. It is neither right, nor very useful, to claim a result when there is still a lot of similarity across the experiences as per the underlying algorithm and the test could go in either direction if run for more time. Hope that helps.

              Terms & ConditionsAccessibility statement

              Loading footer...