What's a good confidence level you're measuring for A/B tests? | Community
Skip to main content
Hamilton_Jie
Hamilton_Jie
Level 2
July 10, 2018
Solved

What's a good confidence level you're measuring for A/B tests?

  • July 10, 2018
  • 2 replies
  • 2611 views

Hey target community,

I recently launched a few A/B tests on our web pages regarding pages' layout and copy. Changes are pretty noticeable if you're a repeat visitor to the page.

However, what I've seen from the data (past few weeks) is that confidence level fluctuates a lot as we're collecting more data. For one test that has been running on a low to medium volume page, we haven't reached statistical significance for any of the challenger experience to either win or lose to the control experience.

I want to ask the community what is the confidence threshold you 're measuring for your web experiment initiatives? Do you have a specific use case where you would take some calculated risks (say an 85% of confidence) but still implementing the experience on the page?

Any thoughts or inputs are much appreciated.

LI

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by Shruti_Premkumar

    Hi Li, Nine Common A/B Testing Pitfalls and How to Avoid Them  and Plan an A/B Test are good resources to consider while planning your tests.

    2 replies

    S
    Adobe Employee
    Shruti_PremkumarAdobe EmployeeAccepted solution
    Adobe Employee
    July 10, 2018

    Hi Li, Nine Common A/B Testing Pitfalls and How to Avoid Them  and Plan an A/B Test are good resources to consider while planning your tests.

      Hamilton_Jie
      Hamilton_JieAuthor
      Level 2
      July 10, 2018

      Thank you. These are great resources. I'll give them a read for sure.

        Terms & ConditionsAccessibility statement

        Loading footer...