SRM Issue with A/B test with 50/50 split (100% audience) | Community
Skip to main content
M
ml19
Level 2
August 8, 2022
Solved

SRM Issue with A/B test with 50/50 split (100% audience)

  • August 8, 2022
  • 2 replies
  • 2962 views

Hi,

I have an experiment I have been working that is showing the SRM issue. 

Is an A/B test with 50/50 split targeting 100% audience.

I relaunched the experiments few times, always trying different setup, but the SRM issue still appears and in some cases, happened 2 weeks after relaunching it.

Any idea of what the cause could be?

 

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by Perrin_Ennen

    It looks like the tool requires an exact 50% to 50% division. As far as I know, this is not possible in Adobe Target. It is always an approximation here. Earlier Math.random() was used for the calculation. In the meantime, window.crypto is used to calculate this. My previous suggestion will not help you here either.

     

    best regards

    2 replies

    Perrin_Ennen
    Community Advisor
    Perrin_EnnenCommunity Advisor
    Community Advisor
    August 9, 2022

    What is the SRM issue? Never had any problems with A/B tests myself so far. @ml19 Maybe you can describe your problem again in more detail.

     

    You can of course create an XT Activity and work with Profile Scripts - if that gets around your problem. Example (profileScript):

    if (mbox.name == 'target-global-mbox') {
if (!user.get('Group_50_50')) {
    var rnd = Math.floor(Math.random() * 100)
    if (rnd <= 49) return 'A';
    if (50 <= rnd) return 'B'
}
}

    best regards

      M
      ml19Author
      Level 2
      August 9, 2022

      Hi @perrin_ennen 

      Thanks for your reply. A Sample Ratio Mismatch (SRM) essentially means that the variants (experiences) of an A/B test have a traffic distribution (sample) that differs in a statistically significant way from what you expected the allocation of traffic or the sample to be.

      In my case, this is the overall data (see screenshot below).

      You can see unique visitors differ considerably. and when i input the unique visitor data into the CXL calculator, it flags an SRM error.

      This is the daily data for the same experiment. You can see the data was fine up until August 1st, which is when the SRM error started appearing

       I can try your solution, but if you have any other idea on how to fix this it will be great!

       

      Thank you!

        Perrin_Ennen
        Community Advisor
        Perrin_EnnenCommunity AdvisorAccepted solution
        Community Advisor
        August 10, 2022

        It looks like the tool requires an exact 50% to 50% division. As far as I know, this is not possible in Adobe Target. It is always an approximation here. Earlier Math.random() was used for the calculation. In the meantime, window.crypto is used to calculate this. My previous suggestion will not help you here either.

         

        best regards

          Amelia_Waliany
          Adobe Employee
          Amelia_WalianyAdobe Employee
          Adobe Employee
          February 13, 2023

          [8.31.22 Adobe Target Community Q&A Coffee Break Answer provided by @mosesmaxen: 

          ----------------------------

          Hi @ml19  - Thank you for your question. If the test is in the initial stage after launch and the numbers are small, you might see an unequal split. However, as the traffic increases the split should be as you have set manually in Target. Example, for a low traffic page (100 Visitors per page) you might not see an equal split initially. The overall period for the split to converge depends on traffic, and not so much on days for which the experiment has been live.  So you may just need to wait a little longer for the data to normalize based on traffic levels.  

           

          Another point to consider, Visitors in an A/B test will stay in their initially-assigned experiences. The traffic change impacts new entrants only.

          ----------------------------

          Check out the entire 8.31.22 Adobe Target Community Q&A Coffee Break discussion thread for more insights on related topics 🙂 

           

            Terms & ConditionsAccessibility statement

            Loading footer...