Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn more

View all

Sign in to view all badges

SOLVED

What's a good confidence level you're measuring for A/B tests?

Hamilton_Jie
Level 2
Level 2

Hey target community,

I recently launched a few A/B tests on our web pages regarding pages' layout and copy. Changes are pretty noticeable if you're a repeat visitor to the page.

However, what I've seen from the data (past few weeks) is that confidence level fluctuates a lot as we're collecting more data. For one test that has been running on a low to medium volume page, we haven't reached statistical significance for any of the challenger experience to either win or lose to the control experience.

I want to ask the community what is the confidence threshold you 're measuring for your web experiment initiatives? Do you have a specific use case where you would take some calculated risks (say an 85% of confidence) but still implementing the experience on the page?

Any thoughts or inputs are much appreciated.

LI

1 Accepted Solution
Shruthi_N
Correct answer by
Employee
Employee

Hi Li, Nine Common A/B Testing Pitfalls and How to Avoid Them  and Plan an A/B Test are good resources to consider while planning your tests.

View solution in original post

2 Replies
Shruthi_N
Correct answer by
Employee
Employee

Hi Li, Nine Common A/B Testing Pitfalls and How to Avoid Them  and Plan an A/B Test are good resources to consider while planning your tests.

View solution in original post

Hamilton_Jie
Level 2
Level 2

Thank you. These are great resources. I'll give them a read for sure.