Interpreting AT test result when the control is winner in one version, and the test is successful in the other version.





I recently ran an A/B test on two versions of the same page - an english version and a chinese version - the results were pushed to Adobe Analytics. The results are the opposite i.e. the chinese version shows that the experience B (experiment) was the winner, whereas in the english version of the test, experience A (control) was the winner. The winner is based on a conversion metric..Point to note: the chinese version of the site receives significantly more traffic and conversion.


My question: 

- In such cases, what are some of the recommended decision rules to consider for determining if the test was successful or not?

- The answer may be that it depends, but what is the thought process or better approach to deciding if the test was successful or not? Thanks!

Accepted Solutions (0)

Answers (2)

Answers (2)





Is conversion metric same for both the Chinese and English version?


If yes , it simply says that visitors of both the versions have different liking.

It may happen that Exp A is a winner in one region and Exp B is a winner in another region, so future decisions should be taken based on the region conversions.


I am referring to regions as I understand that Chinese version is targeting only Chinese audience or region.


Hope that helps to understand!





@bilalhowdoiuseA Can you please confirm if the activity created on Target was an A4T(Analytics for Target) which uses Analytics as a reporting source?

Also, was the metric used in the activity an Analytics metric?

For comparing the results between two solutions, it is best to have integrations(A4T) in place.