Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn more

View all

Sign in to view all badges

SOLVED

Conclude a test with different winners

kathys75015221
Level 3
Level 3

We had some a/b/c/d tests that concluded pretty quickly (normally within a week) with a winner and high confidence level. However, after leaving the test running for another 2 weeks, the confidence level went down and tests concluded with a completely different winner with 95%+ confidence. Has anyone had the same experience and how do you deal with this situation? We have seen the worst performing variation when it's first concluded became the winner when it concluded the second time...

1 Accepted Solution
mikewebguy
Correct answer by
Employee
Employee

kathys75015221

Here is some pertinent information from this KB article: Confidence Level and Confidence Interval

If the confidence level is over 90% or 95%, then the result can be considered statistically significant. Before making any business decisions, try to wait until your sample size is large enough and that the 4 bars of confidence on one or more experiences stays consistent for a continuous length of time to ensure the results are stable.

The following list shows the meaning of the number of confidence bars:

One bar: significance < 60%

Two bars: significance < 75%

Three bars: significance < 90%

Four bars: significance >= 90%

Hope this has been helpful to you!

Mihnea Docea  | Technical Support Rep, Customer Care | Customer & Employee Experience | Adobe | (:: 1 (800) 497-0335

View solution in original post

2 Replies
mikewebguy
Correct answer by
Employee
Employee

kathys75015221

Here is some pertinent information from this KB article: Confidence Level and Confidence Interval

If the confidence level is over 90% or 95%, then the result can be considered statistically significant. Before making any business decisions, try to wait until your sample size is large enough and that the 4 bars of confidence on one or more experiences stays consistent for a continuous length of time to ensure the results are stable.

The following list shows the meaning of the number of confidence bars:

One bar: significance < 60%

Two bars: significance < 75%

Three bars: significance < 90%

Four bars: significance >= 90%

Hope this has been helpful to you!

Mihnea Docea  | Technical Support Rep, Customer Care | Customer & Employee Experience | Adobe | (:: 1 (800) 497-0335

View solution in original post