Level 1

0% to

Level 2

Tip /

to gain points, level up, and earn exciting badges like the new
Mission!

View all

SOLVED

## Does Target account for multiple family errors (Stats) ?

Level 2

Basically what the title says- a statistics question.

For everyone else interested. Multiple family errors (alpha inflation) happens when testing more than one variant to 95% confidence (p = .05). While each test individually has a 5% chance of Type I error, as a family grouping the error rate increased with more variants. (Formula is: 1-(1-alpha)^n where alpha is .05 and n is the number of tests). So if we are testing 8 variations (+ 1 default), the formula would be: 1-(1-.05)^8 = .34 or a 34% chance of seeing a false positive among our grouping if we are testing 8 variants.

Does Target account for this automatically, or do we have to apply a correction post-hoc (or as a priori)?

1 Accepted Solution

Employee

Our sample size calculator offers the ability to account for family-wise error using Bonferroni correction.

We also discuss pitfalls and their resolutions in this article: Nine Common A/B Testing Pitfalls and How to Avoid Them

0 Replies