Adobe Target

ml19 · 8/8/22

Hi,

I have an experiment I have been working that is showing the SRM issue.

Is an A/B test with 50/50 split targeting 100% audience.

I relaunched the experiments few times, always trying different setup, but the SRM issue still appears and in some cases, happened 2 weeks after relaunching it.

Any idea of what the cause could be?

Perrin_Ennen · 8/10/22

It looks like the tool requires an exact 50% to 50% division. As far as I know, this is not possible in Adobe Target. It is always an approximation here. Earlier Math.random() was used for the calculation. In the meantime, window.crypto is used to calculate this. My previous suggestion will not help you here either.

best regards

View solution in original post

Perrin_Ennen · 8/9/22

What is the SRM issue? Never had any problems with A/B tests myself so far. @ml19 Maybe you can describe your problem again in more detail.

You can of course create an XT Activity and work with Profile Scripts - if that gets around your problem. Example (profileScript):

if (mbox.name == 'target-global-mbox') {
if (!user.get('Group_50_50')) {
    var rnd = Math.floor(Math.random() * 100)
    if (rnd <= 49) return 'A';
    if (50 <= rnd) return 'B'
}
}

best regards

ml19 · 8/9/22

Hi @Perrin_Ennen

Thanks for your reply. A Sample Ratio Mismatch (SRM) essentially means that the variants (experiences) of an A/B test have a traffic distribution (sample) that differs in a statistically significant way from what you expected the allocation of traffic or the sample to be.

In my case, this is the overall data (see screenshot below).

10.4 10.4

You can see unique visitors differ considerably. and when i input the unique visitor data into the CXL calculator, it flags an SRM error.

This is the daily data for the same experiment. You can see the data was fine up until August 1st, which is when the SRM error started appearing

10-4

I can try your solution, but if you have any other idea on how to fix this it will be great!

Thank you!

Perrin_Ennen · 8/10/22

It looks like the tool requires an exact 50% to 50% division. As far as I know, this is not possible in Adobe Target. It is always an approximation here. Earlier Math.random() was used for the calculation. In the meantime, window.crypto is used to calculate this. My previous suggestion will not help you here either.

best regards

ml19 · 8/10/22

Thanks @Perrin_Ennen

You have been a great help!

Do you know if is possible to use the profile script with Math.random() and manually assign Control or Variation? Something very similar of the script you shared, but instead activate Control or Variation.

You think is possible?

Amelia_Waliany · 2/13/23

[8.31.22 Adobe Target Community Q&A Coffee Break Answer provided by @mosesmaxen:

----------------------------

Hi @ml19 - Thank you for your question. If the test is in the initial stage after launch and the numbers are small, you might see an unequal split. However, as the traffic increases the split should be as you have set manually in Target. Example, for a low traffic page (100 Visitors per page) you might not see an equal split initially. The overall period for the split to converge depends on traffic, and not so much on days for which the experiment has been live. So you may just need to wait a little longer for the data to normalize based on traffic levels.

Another point to consider, Visitors in an A/B test will stay in their initially-assigned experiences. The traffic change impacts new entrants only.

----------------------------

Check out the entire 8.31.22 Adobe Target Community Q&A Coffee Break discussion thread for more insights on related topics