Conclude a test with different winners

Avatar

Avatar
Validate 1
Level 3
kathys75015221
Level 3

Likes

27 likes

Total Posts

24 posts

Correct reply

0 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 25
Boost 10
View profile

Avatar
Validate 1
Level 3
kathys75015221
Level 3

Likes

27 likes

Total Posts

24 posts

Correct reply

0 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 25
Boost 10
View profile
kathys75015221
Level 3

17-01-2019

We had some a/b/c/d tests that concluded pretty quickly (normally within a week) with a winner and high confidence level. However, after leaving the test running for another 2 weeks, the confidence level went down and tests concluded with a completely different winner with 95%+ confidence. Has anyone had the same experience and how do you deal with this situation? We have seen the worst performing variation when it's first concluded became the winner when it concluded the second time...

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar
Coach
Employee
mikewebguy
Employee

Likes

635 likes

Total Posts

402 posts

Correct reply

85 solutions
Top badges earned
Coach
Contributor
Shape 25
Shape 1
Shape 10
View profile

Avatar
Coach
Employee
mikewebguy
Employee

Likes

635 likes

Total Posts

402 posts

Correct reply

85 solutions
Top badges earned
Coach
Contributor
Shape 25
Shape 1
Shape 10
View profile
mikewebguy
Employee

18-01-2019

kathys75015221

Here is some pertinent information from this KB article: Confidence Level and Confidence Interval

If the confidence level is over 90% or 95%, then the result can be considered statistically significant. Before making any business decisions, try to wait until your sample size is large enough and that the 4 bars of confidence on one or more experiences stays consistent for a continuous length of time to ensure the results are stable.

The following list shows the meaning of the number of confidence bars:

One bar: significance < 60%

Two bars: significance < 75%

Three bars: significance < 90%

Four bars: significance >= 90%

Hope this has been helpful to you!

Mihnea Docea  | Technical Support Rep, Customer Care | Customer & Employee Experience | Adobe | (:: 1 (800) 497-0335

Answers (1)

Answers (1)

Avatar

Avatar
Validate 1
Level 3
kathys75015221
Level 3

Likes

27 likes

Total Posts

24 posts

Correct reply

0 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 25
Boost 10
View profile

Avatar
Validate 1
Level 3
kathys75015221
Level 3

Likes

27 likes

Total Posts

24 posts

Correct reply

0 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 25
Boost 10
View profile
kathys75015221
Level 3

21-01-2019

Thank you Mihnea