50/50 Test Actually 56/44

Avatar

Avatar
Validate 25
Level 4
alim92960278
Level 4

Likes

37 likes

Total Posts

103 posts

Correct reply

5 solutions
Top badges earned
Validate 25
Validate 10
Validate 1
Boost 5
Boost 3
View profile

Avatar
Validate 25
Level 4
alim92960278
Level 4

Likes

37 likes

Total Posts

103 posts

Correct reply

5 solutions
Top badges earned
Validate 25
Validate 10
Validate 1
Boost 5
Boost 3
View profile
alim92960278
Level 4

17-04-2018

Why is it that my 50/50 A/B tests are often lopsided with more Experience A visitors?

Ali

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar
Boost 5
Level 4
russpainter
Level 4

Likes

49 likes

Total Posts

40 posts

Correct reply

7 solutions
Top badges earned
Boost 5
Boost 3
Boost 25
Boost 10
Boost 1
View profile

Avatar
Boost 5
Level 4
russpainter
Level 4

Likes

49 likes

Total Posts

40 posts

Correct reply

7 solutions
Top badges earned
Boost 5
Boost 3
Boost 25
Boost 10
Boost 1
View profile
russpainter
Level 4

17-04-2018

This is basic statistics. If you flip a coin 100 times you could potentially get lopsided heads or lopsided tails. If you continue to flip a coin a million times you should experience a close 50/50 percentage of time and the more times you flip.

You can tell when an a/b test has normalized and the noise of adding a new experience has leveled out when your experiences come back to an even split.

This is the principal behind A/A tests as well. If you run an A/A/B Test you know you can call it when you have significance and the A cells have leveled to 50/50 on traffic and conversion.

Let me know if you have questions

Russ

Answers (1)

Answers (1)

Avatar

Avatar
Validate 25
Level 4
alim92960278
Level 4

Likes

37 likes

Total Posts

103 posts

Correct reply

5 solutions
Top badges earned
Validate 25
Validate 10
Validate 1
Boost 5
Boost 3
View profile

Avatar
Validate 25
Level 4
alim92960278
Level 4

Likes

37 likes

Total Posts

103 posts

Correct reply

5 solutions
Top badges earned
Validate 25
Validate 10
Validate 1
Boost 5
Boost 3
View profile
alim92960278
Level 4

17-04-2018

This makes perfect sense. Thanks,

Ali