Subject Line - A/B Test

Forum|Forum|8 years ago
March 1, 2018
1 reply
3376 views

How do you determine the winner if the criteria is Open Email. If it's based on the count of open email during the testing, can someone explain why on our testing the winner was Subject B but Subject A performed better.

These are the result:

Subject A - Open Email: 303

Subject B - Open Email : 299

Marketo

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

Valerie_Whitin2

Level 4

It is based on the open rate. Which is calculated by # of opens/# delivered. So, it is possible for the B test group to win if it had a smaller delivery group. Hope this helps!

A

Anonymous

Hi Valerie,

I'm having a similar issue. Though even with a smaller delivery group, my B subject line performed worse than A and was still declared the winner. Please see below:

Subject Line A - 29 opens /// 586 delivered = 4.95% open rate

Subject Line B - 24 opens /// 584 delivered = 4.11% open rate

And if it somehow matters, A had 5 clicks while B had 4. It looks like in every way, B performed worse.

Also, the Marketo A/B test winner notification shows B as winning with some absurdly vague "Opens" number:

This notification is the only "solid" reasoning I've found for B winning the test, but I can't figure out in the slightest how this number is calculated -- and I've come up blank in combing the forums thus far.. Can you provide any insight on this?

Thank you!

PJW