Best practice for test sample size on A/B email test | Community
Skip to main content
A
Anonymous
December 31, 2014
Solved

Best practice for test sample size on A/B email test

  • December 31, 2014
  • 2 replies
  • 2900 views
What is a good guideline for size of the test group? My smart list is approximately 80k people. Is 5% (4,000) too low for a test group? Would appreciate any benchmarks or best practices that people have used. Thank you.
This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Josh_Hill13
Alisa,
Typically statistics will provide the correct sample size. A minimum of 30 people is a rule of thumb, but for larger populations like yours, you should calculate the proper sample size.
http://www.surveysystem.com/sscalc.htm

yields 382 at 95% confidence at +/- 5%
if we narrow it to +/- 1% then you need 8575

Generally Marketo's default of 10% should work for most situations. Keep in mind that if you choose 5%, only 2.5% will receive A or B. Thus, you might want to keep the smallest sample at 800-1000.

2 replies

Josh_Hill13
Josh_Hill13Accepted solution
Level 10
December 31, 2014
Alisa,
Typically statistics will provide the correct sample size. A minimum of 30 people is a rule of thumb, but for larger populations like yours, you should calculate the proper sample size.
http://www.surveysystem.com/sscalc.htm

yields 382 at 95% confidence at +/- 5%
if we narrow it to +/- 1% then you need 8575

Generally Marketo's default of 10% should work for most situations. Keep in mind that if you choose 5%, only 2.5% will receive A or B. Thus, you might want to keep the smallest sample at 800-1000.
A
Anonymous
December 31, 2014
Thanks Josh - very helpful!
Terms & ConditionsAccessibility statement

Loading footer...