A/B testing issue | Community
Skip to main content
January 29, 2025
Solved

A/B testing issue

  • January 29, 2025
  • 3 replies
  • 1502 views

Hi Community,

 

I conducted an A/B test using a journey where I created 4 treatments, each with different subject lines and content. I allocated 25% to each treatment and triggered the emails to a test audience. However, despite expecting that each person would receive a different email,  but all 4 participants received the same email and Subject line.

Please anyone help me for this what's wrong? 

@anuhya-y  @mohan_dugganab @davidkangni @vraghav @satheeskannak 

Best answer by RussLewisAdobe

Hi @russlewisadobe ,

 

My question is about the percentage update. It's not working properly or not being distributed correctly when the journey is triggered. All profiles are only receiving 1 or 2 treatment emails, even though I created 4 treatments and tested with 4 profiles. audience distribution also added 25% each. 

Below the  scenario i am testing 

Screenshot-1: 4 different Emails

Screenshot-2: Audience Distribution

Screenshot-3: Profile Count

Result:

The result was that three profiles received email 1 (treatment 1), and one profile received email 4 (treatment 4). No profiles received emails 2 and 3. However, in the end, each profile was supposed to receive only one email, as we aimed to distribute the audience evenly with a 25% split for each treatment, but this distribution didn’t work as planned.

 

 

Hope you understand my question now.

@anuhya-y @mohan_dugganab @davidkangni @vraghav @satheeskannak @tylerkrause  

 


Thanks for clarifying. A few more questions:
1. Is this a read audience or audience qualification/event driven? 

2. Is "allow re-entry" clicked? 

3. Is a customer getting multiple emails here even though you have this set to not allow re-entry and its batch?

 

David is correct that larger sample sizes will represent a more even distribution. The timing doesn't matter as much given that it really is just a random function that will have the same random distribution everytime. 

3 replies

TylerKrause
Adobe Champion
Adobe Champion
January 31, 2025

Sorry to hear about that issue, @ajo_wisdomchase. Can you share some additional details and information on your configuration for the A/B test?

RussLewisAdobe
Adobe Employee
Adobe Employee
January 31, 2025

If you only had 4 participants in the test, it is possible the coin flip placed each person into the same experience 4 times. We randomly assign and with low sample sizes, it is possible to receive the same experience. This is very similar to if you flipped a coin and got heads 4 times in a row. 

Hope this helps!

TylerKrause
Adobe Champion
Adobe Champion
January 31, 2025

If there were 4 treatments, wouldn't that be a 1/256 chance? I believe that would be highly unlikely to be the issue.

Sukrity_Wadhwa
Community Manager
Community Manager
February 21, 2025

Hi @ajo_wisdomchase,

Were you able to come to resolution with the help of the information shared on this thread, or do you still need further assistance? Please let us know. If any of the replies were helpful in moving you closer to a resolution, even partially, we encourage you to mark the one that helped the most as the 'Correct Reply.'
Thank you!

Sukrity Wadhwa