Expand my Community achievements bar.

Applications for the 2024 Adobe Target Community Mentorship Program are open! Click to the right to learn more about participating as either an Aspirant, to professionally level up with a new Certification, or as a Mentor, to share your Adobe Target expertise and inspire through your leadership! Submit your application today.
SOLVED

50/50 Test Actually 56/44

Avatar

Level 5

Why is it that my 50/50 A/B tests are often lopsided with more Experience A visitors?

Ali

1 Accepted Solution

Avatar

Correct answer by
Level 4

This is basic statistics. If you flip a coin 100 times you could potentially get lopsided heads or lopsided tails. If you continue to flip a coin a million times you should experience a close 50/50 percentage of time and the more times you flip.

You can tell when an a/b test has normalized and the noise of adding a new experience has leveled out when your experiences come back to an even split.

This is the principal behind A/A tests as well. If you run an A/A/B Test you know you can call it when you have significance and the A cells have leveled to 50/50 on traffic and conversion.

Let me know if you have questions

Russ

View solution in original post

2 Replies

Avatar

Correct answer by
Level 4

This is basic statistics. If you flip a coin 100 times you could potentially get lopsided heads or lopsided tails. If you continue to flip a coin a million times you should experience a close 50/50 percentage of time and the more times you flip.

You can tell when an a/b test has normalized and the noise of adding a new experience has leveled out when your experiences come back to an even split.

This is the principal behind A/A tests as well. If you run an A/A/B Test you know you can call it when you have significance and the A cells have leveled to 50/50 on traffic and conversion.

Let me know if you have questions

Russ

Avatar

Level 5

This makes perfect sense. Thanks,

Ali