What's a good confidence level you're measuring for A/B tests?

Avatar

Avatar
Springboard
Level 2
Hamilton_Jie
Level 2

Likes

8 likes

Total Posts

10 posts

Correct reply

0 solutions
Top badges earned
Springboard
Establish
Ignite 1
Validate 1
Boost 5
View profile

Avatar
Springboard
Level 2
Hamilton_Jie
Level 2

Likes

8 likes

Total Posts

10 posts

Correct reply

0 solutions
Top badges earned
Springboard
Establish
Ignite 1
Validate 1
Boost 5
View profile
Hamilton_Jie
Level 2

10-07-2018

Hey target community,

I recently launched a few A/B tests on our web pages regarding pages' layout and copy. Changes are pretty noticeable if you're a repeat visitor to the page.

However, what I've seen from the data (past few weeks) is that confidence level fluctuates a lot as we're collecting more data. For one test that has been running on a low to medium volume page, we haven't reached statistical significance for any of the challenger experience to either win or lose to the control experience.

I want to ask the community what is the confidence threshold you 're measuring for your web experiment initiatives? Do you have a specific use case where you would take some calculated risks (say an 85% of confidence) but still implementing the experience on the page?

Any thoughts or inputs are much appreciated.

LI

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar
Give Back 50
Employee
Shruthi_N
Employee

Likes

272 likes

Total Posts

236 posts

Correct reply

64 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile

Avatar
Give Back 50
Employee
Shruthi_N
Employee

Likes

272 likes

Total Posts

236 posts

Correct reply

64 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile
Shruthi_N
Employee

10-07-2018

Hi Li, Nine Common A/B Testing Pitfalls and How to Avoid Them  and Plan an A/B Test are good resources to consider while planning your tests.

Answers (1)

Answers (1)

Avatar

Avatar
Springboard
Level 2
Hamilton_Jie
Level 2

Likes

8 likes

Total Posts

10 posts

Correct reply

0 solutions
Top badges earned
Springboard
Establish
Ignite 1
Validate 1
Boost 5
View profile

Avatar
Springboard
Level 2
Hamilton_Jie
Level 2

Likes

8 likes

Total Posts

10 posts

Correct reply

0 solutions
Top badges earned
Springboard
Establish
Ignite 1
Validate 1
Boost 5
View profile
Hamilton_Jie
Level 2

10-07-2018

Thank you. These are great resources. I'll give them a read for sure.