Why is it that my 50/50 A/B tests are often lopsided with more Experience A visitors?
Ali
Solved! Go to Solution.
This is basic statistics. If you flip a coin 100 times you could potentially get lopsided heads or lopsided tails. If you continue to flip a coin a million times you should experience a close 50/50 percentage of time and the more times you flip.
You can tell when an a/b test has normalized and the noise of adding a new experience has leveled out when your experiences come back to an even split.
This is the principal behind A/A tests as well. If you run an A/A/B Test you know you can call it when you have significance and the A cells have leveled to 50/50 on traffic and conversion.
Let me know if you have questions
Russ
This is basic statistics. If you flip a coin 100 times you could potentially get lopsided heads or lopsided tails. If you continue to flip a coin a million times you should experience a close 50/50 percentage of time and the more times you flip.
You can tell when an a/b test has normalized and the noise of adding a new experience has leveled out when your experiences come back to an even split.
This is the principal behind A/A tests as well. If you run an A/A/B Test you know you can call it when you have significance and the A cells have leveled to 50/50 on traffic and conversion.
Let me know if you have questions
Russ
This makes perfect sense. Thanks,
Ali
Views
Replies
Total Likes