Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn more

View all

Sign in to view all badges

SOLVED

50/50 Test Actually 56/44

alim92960278
Level 5
Level 5

Why is it that my 50/50 A/B tests are often lopsided with more Experience A visitors?

Ali

1 Accepted Solution
russpainter
Correct answer by
Level 4
Level 4

This is basic statistics. If you flip a coin 100 times you could potentially get lopsided heads or lopsided tails. If you continue to flip a coin a million times you should experience a close 50/50 percentage of time and the more times you flip.

You can tell when an a/b test has normalized and the noise of adding a new experience has leveled out when your experiences come back to an even split.

This is the principal behind A/A tests as well. If you run an A/A/B Test you know you can call it when you have significance and the A cells have leveled to 50/50 on traffic and conversion.

Let me know if you have questions

Russ

View solution in original post

2 Replies
russpainter
Correct answer by
Level 4
Level 4

This is basic statistics. If you flip a coin 100 times you could potentially get lopsided heads or lopsided tails. If you continue to flip a coin a million times you should experience a close 50/50 percentage of time and the more times you flip.

You can tell when an a/b test has normalized and the noise of adding a new experience has leveled out when your experiences come back to an even split.

This is the principal behind A/A tests as well. If you run an A/A/B Test you know you can call it when you have significance and the A cells have leveled to 50/50 on traffic and conversion.

Let me know if you have questions

Russ

View solution in original post