Which criteria does Marketo use to divide the audience into A and B? | Community
Skip to main content
Level 2
December 14, 2020
Solved

Which criteria does Marketo use to divide the audience into A and B?

  • December 14, 2020
  • 1 reply
  • 3987 views

I've done an A/B  test using the whole email variation test type with an audience of 3000 people. The A/B test was done on everyone (I’ve selected 100% thus I haven’t selected any winner criteria). So far 1 email got 45% more opens than the other (that’s about 60 people). I’m confused because the subject and preview text are the same… the low opens one even had slightly more delivered (10+ people) than the other one.

 

The audience for the test is big enough to make a statistically significant conclusion that the subject line or/and preview text in As is much better than in B however it’s not the case since both are the same.

 

Any idea why that would happen?

 

Which criteria does Marketo use to divide the audience into A and B?


How Marketo makes sure that audience A has the same attributes as the audience B?

 

I feel like Marketo didn’t divide the audiences “fairly”

 

My issue now is how can I rely on Marketo to run the A/B test for us?

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by SanfordWhiteman

The audience for the test is big enough to make a statistically significant conclusion that the subject line or/and preview text in As is much better than in B however it’s not the case since both are the same.

The distribution is random, there's not expected to be any clustering/unclustering by attribute values.

1 reply

SanfordWhiteman
SanfordWhitemanAccepted solution
Level 10
December 15, 2020

The audience for the test is big enough to make a statistically significant conclusion that the subject line or/and preview text in As is much better than in B however it’s not the case since both are the same.

The distribution is random, there's not expected to be any clustering/unclustering by attribute values.

Andy2023Author
Level 2
December 15, 2020

Knowing this how do you run A/B tests in Marketo to produce accurate results?

SanfordWhiteman
Level 10
December 15, 2020

"accurate" is too vague, exactly what distribution algorithm are you expecting?