Does Target account for multiple family errors (Stats) ?

csanderson99 23-06-2017

Basically what the title says- a statistics question.

For everyone else interested. Multiple family errors (alpha inflation) happens when testing more than one variant to 95% confidence (p = .05). While each test individually has a 5% chance of Type I error, as a family grouping the error rate increased with more variants. (Formula is: 1-(1-alpha)^n where alpha is .05 and n is the number of tests). So if we are testing 8 variations (+ 1 default), the formula would be: 1-(1-.05)^8 = .34 or a 34% chance of seeing a false positive among our grouping if we are testing 8 variants.

Does Target account for this automatically, or do we have to apply a correction post-hoc (or as a priori)?

Accepted Solutions (1)

Accepted Solutions (1)

Answers (1)

Answers (1)

csanderson99 23-06-2017

Thanks Jason, so it isn't automatically calculated. That's not an issue, just didn't want to create more work for myself. Do you know what type of Bonferroni correction the calculator uses?