Required amount of traffic for AB test percentages to match up to those defined in activity | Community
Skip to main content
P
philipk92120636
Level 4
May 21, 2021
Solved

Required amount of traffic for AB test percentages to match up to those defined in activity

  • May 21, 2021
  • 1 reply
  • 1431 views

Hi everyone!

We've just deployed an A/B test with a simple 50/50 split.

The test has been running for 3 days and the current split is 66% control, 34% test for 222 visitors (we're using Analytics for Target as the reporting source). Yesterday, it was an 80/20 split between the control and test variants.

The test itself deploys a client side redirect if the customer has a specific cookie value (we're not using the built in redirect to page option in Target as we need to modify the cookie value before applying the redirect). The audience is based on customers hitting the site with a specific query parameter value.

There haven't been any changes since activation (i.e. pausing and reactivated, changes to audience, etc.) so I'm just wondering if there's a ballpark figure where we can expect the numbers to even out as required.

Any information would be greatly appreciated.

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by Amelia_Waliany

    [8.3.22 Adobe Target Community Q&A Coffee Break Answer provided by @rbooth115: 

    ----------------------------

    Hi @philipk92120636 ! Did you use the Target Sample Size Calculator to estimate how long this test needs to run before it reaches significance? I recommend that you don't analyze the results of an A/B tests until it has reached significance (ie: ran for X number of days as defined by the calculator in order to reach significance). Until then, the data might not make sense. 

    Further, please note that a 50/50 traffic split does not mean that every other visitor is sent to the control experience, rather it is a flip of the coin every time, thus it will take a certain amount of site traffic and time for this to even out.

    ----------------------------

    Check out the entire 8.3.22 Adobe Target Community Q&A Coffee Break discussion thread for more insights on related topics 🙂 

    1 reply

    Amelia_Waliany
    Adobe Employee
    Amelia_WalianyAdobe EmployeeAccepted solution
    Adobe Employee
    February 13, 2023

    [8.3.22 Adobe Target Community Q&A Coffee Break Answer provided by @rbooth115: 

    ----------------------------

    Hi @philipk92120636 ! Did you use the Target Sample Size Calculator to estimate how long this test needs to run before it reaches significance? I recommend that you don't analyze the results of an A/B tests until it has reached significance (ie: ran for X number of days as defined by the calculator in order to reach significance). Until then, the data might not make sense. 

    Further, please note that a 50/50 traffic split does not mean that every other visitor is sent to the control experience, rather it is a flip of the coin every time, thus it will take a certain amount of site traffic and time for this to even out.

    ----------------------------

    Check out the entire 8.3.22 Adobe Target Community Q&A Coffee Break discussion thread for more insights on related topics 🙂 

    Terms & ConditionsAccessibility statement

    Loading footer...