I recently ran an A/B test on two versions of the same page - an english version and a chinese version - the results were pushed to Adobe Analytics. The results are the opposite i.e. the chinese version shows that the experience B (experiment) was the winner, whereas in the english version of the test, experience A (control) was the winner. The winner is based on a conversion metric..Point to note: the chinese version of the site receives significantly more traffic and conversion.
- In such cases, what are some of the recommended decision rules to consider for determining if the test was successful or not?
- The answer may be that it depends, but what is the thought process or better approach to deciding if the test was successful or not? Thanks!