What's your sample size? | Adobe Higher Education
Skip to main content
Jeff_Canada2
Level 3
March 6, 2015
Pregunta

What's your sample size?

  • March 6, 2015
  • 3 respuestas
  • 1699 visualizaciones
I am curious to see what the Marketing Nation thinks is an appropriate number of leads for an A/B test? 100? 1,000? 10,000? 2?

3 respuestas

March 6, 2015
I'd say that it depends on the variation you see.  If you send an email to 100 people and 40 open version A and 2 open version B, that's pretty signifcantly in favor of version A.

In general though, I try to test at least a few thousand - 5k-10k just to get a larger sample size and have a more accurate test.

I just use a calculator like this one, and plug in the numbers.
Josh_Hill13
Level 10
March 6, 2015
Yes, use the calculator. There is no "best practice" other than to do the math.

Now if you want to not do math, they say that n=30 is a minimum. In most cases with email, you need several hundred out of your list to have any reasonable significance...so use the calculator.
Michelle_Tizian
Level 10
March 6, 2015
I prefer a minimum of 1000 to 5000 leads for an A/B test. A couple of times I was asked to split a list evenly 50/50 sending 2 different versions of an email to 200 and another time 500 people, it didn't make that big of a difference on the results.  I just found that one list had more bad email address than the other list, but had about the same results for the unsubscribes and CTR.  I also did a few A/B tests on sponsored emails using 10-20% out of 20 to 30k leads and the results were more obvious. I like the idea of using the calculator.