Required amount of traffic for AB test percentages to match up to those defined in activity

[philipk92120636]

Hi everyone!

We've just deployed an A/B test with a simple 50/50 split.

The test has been running for 3 days and the current split is 66% control, 34% test for 222 visitors (we're using Analytics for Target as the reporting source). Yesterday, it was an 80/20 split between the control and test variants.

The test itself deploys a client side redirect if the customer has a specific cookie value (we're not using the built in redirect to page option in Target as we need to modify the cookie value before applying the redirect). The audience is based on customers hitting the site with a specific query parameter value.

There haven't been any changes since activation (i.e. pausing and reactivated, changes to audience, etc.) so I'm just wondering if there's a ballpark figure where we can expect the numbers to even out as required.

Any information would be greatly appreciated.

[Amelia_Waliany · Accepted solution]

[8.3.22 Adobe Target Community Q&A Coffee Break Answer provided by @rbooth115: 

----------------------------

Hi @philipk92120636 ! Did you use the Target Sample Size Calculator to estimate how long this test needs to run before it reaches significance? I recommend that you don't analyze the results of an A/B tests until it has reached significance (ie: ran for X number of days as defined by the calculator in order to reach significance). Until then, the data might not make sense. 

Further, please note that a 50/50 traffic split does not mean that every other visitor is sent to the control experience, rather it is a flip of the coin every time, thus it will take a certain amount of site traffic and time for this to even out.

----------------------------

Check out the entire 8.3.22 Adobe Target Community Q&A Coffee Break discussion thread for more insights on related topics  

[Amelia_Waliany · Accepted solution]

[8.3.22 Adobe Target Community Q&A Coffee Break Answer provided by @rbooth115: 

----------------------------

Hi @philipk92120636 ! Did you use the Target Sample Size Calculator to estimate how long this test needs to run before it reaches significance? I recommend that you don't analyze the results of an A/B tests until it has reached significance (ie: ran for X number of days as defined by the calculator in order to reach significance). Until then, the data might not make sense. 

Further, please note that a 50/50 traffic split does not mean that every other visitor is sent to the control experience, rather it is a flip of the coin every time, thus it will take a certain amount of site traffic and time for this to even out.

----------------------------

Check out the entire 8.3.22 Adobe Target Community Q&A Coffee Break discussion thread for more insights on related topics  

[Amelia_Waliany · Accepted solution]

[8.3.22 Adobe Target Community Q&A Coffee Break Answer provided by @rbooth115: 

----------------------------

Hi @philipk92120636 ! Did you use the Target Sample Size Calculator to estimate how long this test needs to run before it reaches significance? I recommend that you don't analyze the results of an A/B tests until it has reached significance (ie: ran for X number of days as defined by the calculator in order to reach significance). Until then, the data might not make sense. 

Further, please note that a 50/50 traffic split does not mean that every other visitor is sent to the control experience, rather it is a flip of the coin every time, thus it will take a certain amount of site traffic and time for this to even out.

----------------------------

Check out the entire 8.3.22 Adobe Target Community Q&A Coffee Break discussion thread for more insights on related topics