Conclude a test with different winners



We had some a/b/c/d tests that concluded pretty quickly (normally within a week) with a winner and high confidence level. However, after leaving the test running for another 2 weeks, the confidence level went down and tests concluded with a completely different winner with 95%+ confidence. Has anyone had the same experience and how do you deal with this situation? We have seen the worst performing variation when it's first concluded became the winner when it concluded the second time...

View Entire Topic