Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
Bedrock Mission!

Learn more

View all

Sign in to view all badges

SRM Issue with A/B test with 50/50 split (100% audience)

Avatar

Level 2

Hi,

I have an experiment I have been working that is showing the SRM issue. 

Is an A/B test with 50/50 split targeting 100% audience.

I relaunched the experiments few times, always trying different setup, but the SRM issue still appears and in some cases, happened 2 weeks after relaunching it.

Any idea of what the cause could be?

 

3 Replies

Avatar

Level 4

What is the SRM issue? Never had any problems with A/B tests myself so far. @ml19 Maybe you can describe your problem again in more detail.

 

You can of course create an XT Activity and work with Profile Scripts - if that gets around your problem. Example (profileScript):

if (mbox.name == 'target-global-mbox') {
if (!user.get('Group_50_50')) {
    var rnd = Math.floor(Math.random() * 100)
    if (rnd <= 49) return 'A';
    if (50 <= rnd) return 'B'
}
}

best regards

Avatar

Level 2

Hi @perrin_ennen 

Thanks for your reply. A Sample Ratio Mismatch (SRM) essentially means that the variants (experiences) of an A/B test have a traffic distribution (sample) that differs in a statistically significant way from what you expected the allocation of traffic or the sample to be.

In my case, this is the overall data (see screenshot below).

10.410.4

You can see unique visitors differ considerably. and when i input the unique visitor data into the CXL calculator, it flags an SRM error.

This is the daily data for the same experiment. You can see the data was fine up until August 1st, which is when the SRM error started appearing

10-410-4

 I can try your solution, but if you have any other idea on how to fix this it will be great!

 

Thank you!

Avatar

Level 4

It looks like the tool requires an exact 50% to 50% division. As far as I know, this is not possible in Adobe Target. It is always an approximation here. Earlier Math.random() was used for the calculation. In the meantime, window.crypto is used to calculate this. My previous suggestion will not help you here either.

 

best regards

Avatar

Level 2

Thanks @perrin_ennen 

You have been a great help!

Do you know if is possible to use the profile script with Math.random() and manually assign Control or Variation? Something very similar of the script you shared, but instead activate Control or Variation.

You think is possible?