Interpreting AT test result when the control is winner in one version, and the test is successful in the other version.

[#1]

Hi, 

I recently ran an A/B test on two versions of the same page - an english version and a chinese version - the results were pushed to Adobe Analytics. The results are the opposite i.e. the chinese version shows that the experience B (experiment) was the winner, whereas in the english version of the test, experience A (control) was the winner. The winner is based on a conversion metric..Point to note: the chinese version of the site receives significantly more traffic and conversion.

My question: 

- In such cases, what are some of the recommended decision rules to consider for determining if the test was successful or not?

- The answer may be that it depends, but what is the thought process or better approach to deciding if the test was successful or not? Thanks!

[bilalhowdoiuseA · Accepted solution]

@bilalhowdoiuseA 

Is conversion metric same for both the Chinese and English version?

If yes , it simply says that visitors of both the versions have different liking.

It may happen that Exp A is a winner in one region and Exp B is a winner in another region, so future decisions should be taken based on the region conversions.

I am referring to regions as I understand that Chinese version is targeting only Chinese audience or region.

Hope that helps to understand!

[bilalhowdoiuseA · Accepted solution]

Thank you for the answer - i should have been more clear, my mistake.

[bilalhowdoiuseA · Accepted solution]

Thank you for the answer - i should have been more clear, my mistake. 1- the conversion metric is the same for both versions of the test 2- the test is in the same country (Hong Kong) and the website offers both languages for visitors, with the Chinese version of the page having significantly higher traffic. The interesting dilemma is wether to deploy the winning version of the test on BOTH the versions, as the experience has to be the same, however the English visitors did not show a positive response to the test

[TanviAgarwal89 · Accepted solution]

Thankyou. This is a tricky business decision . If I would have been you and wish to keep the same experience on both English and Chinese version, then I would have gone with Exp B as it is the overall winner of the activity. Also this is aligned with the chinese version which have more traffic.

[TanviAgarwal89 · Accepted solution]

Else you can also explore the options of creating the experience based on the language selection, but this would definitely have different layouts on both the En and Ch versions.

However if you are creating an AB activity , this is how it will work I believe

Visitor lands on English version > views exp A 

Now moves to the Chinese version > he will still remain in experience A (as experience will not be reevaluated again)

Yes you would need to create 2 different activities in this case and explore the options. 🙂

[bilalhowdoiuseA · Accepted solution]

Makes sense - i agree, i would weight the version with the higher traffic more heavily and implement Exp B universally. Thanks!

[bilalhowdoiuseA]

@bilalhowdoiuseA Can you please confirm if the activity created on Target was an A4T(Analytics for Target) which uses Analytics as a reporting source?

Also, was the metric used in the activity an Analytics metric?

For comparing the results between two solutions, it is best to have integrations(A4T) in place.

Regards,

Karan

[bilalhowdoiuseA]

Yes, the activity was created on A4T and we used a custom analytics metric (form submissions) to compare.