Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

Under what scenario would you not split 50-50?

Avatar

Avatar
Validate 1
Level 5
archits30249934
Level 5

Likes

116 likes

Total Posts

15 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 50
Boost 5
Boost 3
Boost 25
View profile

Avatar
Validate 1
Level 5
archits30249934
Level 5

Likes

116 likes

Total Posts

15 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 50
Boost 5
Boost 3
Boost 25
View profile
archits30249934
Level 5

16-04-2018

I would like to understand from other Target or optimization experts if you have worked on an A/B test where you have not split 50-50 traffic between the control and the challenger and if yes what's the reason behind it.

I understand that doing a 50-50 split reach the statistical significance faster thus reducing the testing period.

What if I do a 20-80 or 80-20 split? If I do that how to know how much traffic do I need per variant to reach statistical significance?

Thanks,

Archit

Replies

Avatar

Avatar
Boost 5
Level 4
russpainter
Level 4

Likes

49 likes

Total Posts

40 posts

Correct Reply

7 solutions
Top badges earned
Boost 5
Boost 3
Boost 25
Boost 10
Boost 1
View profile

Avatar
Boost 5
Level 4
russpainter
Level 4

Likes

49 likes

Total Posts

40 posts

Correct Reply

7 solutions
Top badges earned
Boost 5
Boost 3
Boost 25
Boost 10
Boost 1
View profile
russpainter
Level 4

16-04-2018

The majority of the time I do an equal split, but here are some cases when I do 90/10 or 80/20. There are a bunch of confidence calculators that will tell you how long something should be tested with an unequal split.

1. 90/10 -  Called a "Hold-out" Where I plan on rolling a new feature or creative out with full on a/b testing it. These would be things that have to happen due to technology upgrades or site standards pushes. They basically provide a baseline for how the new feature is performing against default. Determining if the new feature is hurting conversion.

Also do 90/10 splits when ramping a new product feature and making sure that the new feature can handle the server request loads. 

In both of these scenarios, usually don't wait for significance. They have directional data only.

2. 80/20 - Similar to a hold-out I've run 80/20 when running a price test or a test that could strongly affect the business if it lost. Starting out will displaying 20% to new pricing and package mixes and see how they perform against the default. This is just used as a safe-guard. If everything checks out, you could then ramp up to an even 50/50 and wait for significance.

I've seen these types of distributions for promotional banners and discount codes as well. Making the discount or promotion more exclusive since it is a smaller percentage that would see it. 

There are a lot of different reasons that could be applied depending on what you are trying to accomplish for the needs of the company.

Avatar

Avatar
Contributor
Level 4
rajneesh_gautam
Level 4

Likes

64 likes

Total Posts

68 posts

Correct Reply

12 solutions
Top badges earned
Contributor
Ignite 5
Ignite 3
Ignite 1
Validate 1
View profile

Avatar
Contributor
Level 4
rajneesh_gautam
Level 4

Likes

64 likes

Total Posts

68 posts

Correct Reply

12 solutions
Top badges earned
Contributor
Ignite 5
Ignite 3
Ignite 1
Validate 1
View profile
rajneesh_gautam
Level 4

17-04-2018

In addition to Russ' response, another reason why we use 90/10 or 80/20 testing is to create baseline-audience while running Personalisation campaigns. In these scenarios, 90% or 80% of the traffic is shown personalised experience while rest are shown default experience. This way, it becomes easier for us to assess incremental business benefit of personalisation campaigns.

Hope this helps,

Rajneesh

Avatar

Avatar
Give Back 3
Level 2
celinab5183000
Level 2

Likes

3 likes

Total Posts

4 posts

Correct Reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile

Avatar
Give Back 3
Level 2
celinab5183000
Level 2

Likes

3 likes

Total Posts

4 posts

Correct Reply

0 solutions
Top badges earned
Give Back 3
Give Back
Ignite 1
Boost 3
Boost 1
View profile
celinab5183000
Level 2

22-10-2019

Can you give some links to the "bunch of confidence calculators" that will help calculate traffic needed for an uneven traffic split?