A/B testing issue

[Ajo_WisdomChase]

Hi Community,

I conducted an A/B test using a journey where I created 4 treatments, each with different subject lines and content. I allocated 25% to each treatment and triggered the emails to a test audience. However, despite expecting that each person would receive a different email,  but all 4 participants received the same email and Subject line.

Please anyone help me for this what's wrong? 

@Anuhya-Y  @Mohan_Dugganab @DavidKangni @vraghav @SatheeskannaK 

[TylerKrause]

Sorry to hear about that issue, @Ajo_WisdomChase. Can you share some additional details and information on your configuration for the A/B test?

[RussLewisAdobe]

If you only had 4 participants in the test, it is possible the coin flip placed each person into the same experience 4 times. We randomly assign and with low sample sizes, it is possible to receive the same experience. This is very similar to if you flipped a coin and got heads 4 times in a row. 

Hope this helps!

[TylerKrause]

If there were 4 treatments, wouldn't that be a 1/256 chance? I believe that would be highly unlikely to be the issue.

[Ajo_WisdomChase]

Hi @RussLewisAdobe ,

If we are splitting randomly, what is the purpose of using this percentage split? what is use of Audience distribution?

[RussLewisAdobe]

So I think you’re asking what’s the difference between content experimentations audience distribution and the percent split via a condition on the journey canvas. Is that right?

content experimentations come with ootb reporting for lift, confidence etc. 

condition splits on canvas do not and require manual analysis after the journey is run. 

the random function to distribute profiles I believe is the same in both. 

[Ajo_WisdomChase]

Hi @RussLewisAdobe ,

My question is about the percentage update. It's not working properly or not being distributed correctly when the journey is triggered. All profiles are only receiving 1 or 2 treatment emails, even though I created 4 treatments and tested with 4 profiles. audience distribution also added 25% each. 

Below the  scenario i am testing 

Screenshot-1: 4 different Emails

Screenshot-2: Audience Distribution

Screenshot-3: Profile Count

Result:

The result was that three profiles received email 1 (treatment 1), and one profile received email 4 (treatment 4). No profiles received emails 2 and 3. However, in the end, each profile was supposed to receive only one email, as we aimed to distribute the audience evenly with a 25% split for each treatment, but this distribution didn’t work as planned.

Hope you understand my question now.

@Anuhya-Y @Mohan_Dugganab @DavidKangni @vraghav @SatheeskannaK @TylerKrause  

[DavidKangni]

@Ajo_WisdomChase 

Can you try the same in a Campaign? AS @RussLewisAdobe  mentioned, in a journey the distribution can possibly assign the same experience.

The reason is the timestamp the profiles reach your activity. Basically If the 4 profiles reach at the same time then high chance to have the correct distribution

If 3 profiles reach the same time but 1 profile reach few milliseconds after then you may not have a 25% distribution.

I will recommend to use a higher volume for testing as the confidence calculation used will be biased with this lower volume.

Hope it helps.

Thanks,

David

[RussLewisAdobe]

Thanks for clarifying. A few more questions:
1. Is this a read audience or audience qualification/event driven? 

2. Is "allow re-entry" clicked? 

3. Is a customer getting multiple emails here even though you have this set to not allow re-entry and its batch?

David is correct that larger sample sizes will represent a more even distribution. The timing doesn't matter as much given that it really is just a random function that will have the same random distribution everytime. 

[Sukrity_Wadhwa]

Hi @Ajo_WisdomChase,

Were you able to come to resolution with the help of the information shared on this thread, or do you still need further assistance? Please let us know. If any of the replies were helpful in moving you closer to a resolution, even partially, we encourage you to mark the one that helped the most as the 'Correct Reply.'
Thank you!