Interpreting AT test result when the control is winner in one version, and the test is successful in the other version.

bilalhowdoiuseA 05-05-2020



I recently ran an A/B test on two versions of the same page - an english version and a chinese version - the results were pushed to Adobe Analytics. The results are the opposite i.e. the chinese version shows that the experience B (experiment) was the winner, whereas in the english version of the test, experience A (control) was the winner. The winner is based on a conversion metric..Point to note: the chinese version of the site receives significantly more traffic and conversion.


My question: 

- In such cases, what are some of the recommended decision rules to consider for determining if the test was successful or not?

- The answer may be that it depends, but what is the thought process or better approach to deciding if the test was successful or not? Thanks!

Mark Solution

Are these answers useful?
Help other community members by marking useful answers as accepted.

Accepted Solutions (0)

Answers (2)

Answers (2)

TanviAgarwal89 20-05-2020



Is conversion metric same for both the Chinese and English version?


If yes , it simply says that visitors of both the versions have different liking.

It may happen that Exp A is a winner in one region and Exp B is a winner in another region, so future decisions should be taken based on the region conversions.


I am referring to regions as I understand that Chinese version is targeting only Chinese audience or region.


Hope that helps to understand!



karand53490614 20-05-2020

@bilalhowdoiuseA Can you please confirm if the activity created on Target was an A4T(Analytics for Target) which uses Analytics as a reporting source?

Also, was the metric used in the activity an Analytics metric?

For comparing the results between two solutions, it is best to have integrations(A4T) in place.