Level 1

0% to

Level 2

Tip /

to gain points, level up, and earn exciting badges like the new
Mission!

View all

## Does Target account for multiple family errors (Stats) ?

csanderson99
Level 2

Like

1 like

Total Posts

12 posts

0 solutions
View profile

csanderson99
Level 2

Like

1 like

Total Posts

12 posts

0 solutions
View profile
csanderson99
Level 2

23-06-2017

Basically what the title says- a statistics question.

For everyone else interested. Multiple family errors (alpha inflation) happens when testing more than one variant to 95% confidence (p = .05). While each test individually has a 5% chance of Type I error, as a family grouping the error rate increased with more variants. (Formula is: 1-(1-alpha)^n where alpha is .05 and n is the number of tests). So if we are testing 8 variations (+ 1 default), the formula would be: 1-(1-.05)^8 = .34 or a 34% chance of seeing a false positive among our grouping if we are testing 8 variants.

Does Target account for this automatically, or do we have to apply a correction post-hoc (or as a priori)?

#### Accepted Solutions (1)

Jason_Hickey
Employee

Likes

44 likes

Total Posts

32 posts

3 solutions
View profile

Jason_Hickey
Employee

Likes

44 likes

Total Posts

32 posts

3 solutions
View profile
Jason_Hickey
Employee

23-06-2017

Our sample size calculator offers the ability to account for family-wise error using Bonferroni correction.

We also discuss pitfalls and their resolutions in this article: Nine Common A/B Testing Pitfalls and How to Avoid Them

csanderson99
Level 2

Like

1 like

Total Posts

12 posts

0 solutions
View profile

csanderson99
Level 2

Like

1 like

Total Posts

12 posts