Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

Does Target account for multiple family errors (Stats) ?

Avatar

Avatar
Validate 1
Level 2
csanderson99
Level 2

Like

1 like

Total Posts

12 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 1
View profile

Avatar
Validate 1
Level 2
csanderson99
Level 2

Like

1 like

Total Posts

12 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 1
View profile
csanderson99
Level 2

23-06-2017

Basically what the title says- a statistics question.

For everyone else interested. Multiple family errors (alpha inflation) happens when testing more than one variant to 95% confidence (p = .05). While each test individually has a 5% chance of Type I error, as a family grouping the error rate increased with more variants. (Formula is: 1-(1-alpha)^n where alpha is .05 and n is the number of tests). So if we are testing 8 variations (+ 1 default), the formula would be: 1-(1-.05)^8 = .34 or a 34% chance of seeing a false positive among our grouping if we are testing 8 variants.

Does Target account for this automatically, or do we have to apply a correction post-hoc (or as a priori)?

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar
Ignite 1
Employee
Jason_Hickey
Employee

Likes

44 likes

Total Posts

32 posts

Correct Reply

3 solutions
Top badges earned
Ignite 1
Give Back 5
Give Back 3
Give Back 10
Give Back
View profile

Avatar
Ignite 1
Employee
Jason_Hickey
Employee

Likes

44 likes

Total Posts

32 posts

Correct Reply

3 solutions
Top badges earned
Ignite 1
Give Back 5
Give Back 3
Give Back 10
Give Back
View profile
Jason_Hickey
Employee

23-06-2017

Our sample size calculator offers the ability to account for family-wise error using Bonferroni correction. 

We also discuss pitfalls and their resolutions in this article: Nine Common A/B Testing Pitfalls and How to Avoid Them

https://docs.adobe.com/content/target-microsite/testcalculator.html

1236771_pastedImage_1.png

Answers (1)

Answers (1)

Avatar

Avatar
Validate 1
Level 2
csanderson99
Level 2

Like

1 like

Total Posts

12 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 1
View profile

Avatar
Validate 1
Level 2
csanderson99
Level 2

Like

1 like

Total Posts

12 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 1
View profile
csanderson99
Level 2

23-06-2017

Thanks Jason, so it isn't automatically calculated. That's not an issue, just didn't want to create more work for myself. Do you know what type of Bonferroni correction the calculator uses?