A/B Testing - Different Send Volumes | Community
Skip to main content
Zack_Bakht
Zack_Bakht
Level 2
May 6, 2019
Question

A/B Testing - Different Send Volumes

  • May 6, 2019
  • 1 reply
  • 4969 views

Hello,

We have had a few different email programs send a different amount of emails to our A variant and our B variant, and was curious to see if anyone knows why that might be happening. For example, a recent date/time test we ran had 67,000 emails delivered to our A variant, and 41,000 emails delivered to our B variant. This one specifically was a morning vs. afternoon send; this set of leads received their most recent email before this all at the same and day, leading me to believe our communication limits are not the issue. Any ideas would help.

Thanks!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

1 reply

Priyank_Joshi3
Priyank_Joshi3
Level 5
May 6, 2019

Hi Zack,

Can you please share the A/B test setup that you have used on your email programs? Per my understanding Date/Time A/B test might have been setup in a way that 60% records receives email version-A while rest of the 40% receives email version-B on defined time. For your reference please refer product doc - Use "Date/Time" A/B Testing - Marketo Docs - Product Documentation

Thanks!

Priyank

Zack_Bakht
Zack_BakhtAuthor
Level 2
May 6, 2019

Hi Priyank,


Thanks for the reply. The test was setup as follows:

Variant A: Morning Send - May 1, 2019, 8:00 AM PDT

Variant B: Afternoon Send - May 1, 2019, 3:00 PM PDT

Test Sample Size: 50%

Winner Criteria: Opens

The Morning Send received 67k emails, Afternoon Send 41k. My understanding is that 50% of the available leads should have received the test, with each variant getting half of the available leads. My confusion is to why it is skewed in favor of the Morning Send; I figured if comm. limits were the issue, the Afternoon Send would be the higher number, due to more leads becoming available as the day passed.

Thanks,

-Zack

Kevin_McMahon1
Kevin_McMahon1
Level 4
May 6, 2019

Was there a separate email that went out between the morning and evening that could have affected part of this audience's communication limits?

Terms & ConditionsAccessibility statement

Loading footer...