A/B Test Limitations | Community
Skip to main content
Craig_Wu
Craig_Wu
Level 2
February 6, 2018
Question

A/B Test Limitations

  • February 6, 2018
  • 1 reply
  • 4325 views

Hi,

We wish to conduct an A/B Whole Email test with 16 versions. Is there any limitations on the number of versions that can be sent in any given test? What are some considerations to factor prior to sending this test?

Thanks!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

1 reply

Dan_Stevens_
Dan_Stevens_
Level 10
February 6, 2018

Do you have an extremely large sample size?  I only ask because it may be difficult to evaluate the most successful email with such a large number of test variants.  Even with a large sample size, I suspect it will still be difficult to understand what elements/content resulted in better conversions.  16 is a lot.

Craig_Wu
Craig_WuAuthor
Level 2
February 6, 2018

Hi Dan,

Thanks for the quick reply. The size is not particularly large: 46,600. I understand that 16 versions is quite a lot. I didn't plan this test, just trying to make sure that it will work as requested and to mitigate any technical problems that we may encounter before we send.

Thanks again for any additional advice on technical concerns.

Craig

Dan_Stevens_
Dan_Stevens_
Level 10
February 6, 2018

Craig - I believe you're limited to only two emails when testing "whole emails".  Unlike other tests - like "subject line" testing - there's not "+" icon to add more test variants.

Terms & ConditionsAccessibility statement

Loading footer...