Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

Adobe Target A/B Test

Avatar

Avatar
Validate 1
Level 2
nehak44235668
Level 2

Likes

7 likes

Total Posts

9 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
View profile

Avatar
Validate 1
Level 2
nehak44235668
Level 2

Likes

7 likes

Total Posts

9 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
View profile
nehak44235668
Level 2

29-08-2019

Hello everyone,

We are currently running an A4T A/B test. This test was launch on August 8th but there was no data coming into Analytics, nor Target until August 20th which is the day the issue was fixed. We are now seeing data in Analytics and Target. Also, we'd asked the Adobe client care team to provide us the data while it was not coming into Target. So now we have data from three different platforms: Analytics, Target and the backend database (that the client care provided). I know that the database data will not match Analytics and Target but they should at least show the same trend. Target show B is the winner whereas Analytics shows A as the winner. Not sure which platform to trust. Has anyone ever been in a similar situation? I'd love to hear your thoughts/ recommendations. We could stop the test and start with a clean slate but again what's the guarantee that the data will match?

Thank you.

Neha

Replies

Avatar

Avatar
Coach
Employee
mikewebguy
Employee

Likes

630 likes

Total Posts

402 posts

Correct Reply

85 solutions
Top badges earned
Coach
Contributor
Shape 25
Shape 1
Shape 10
View profile

Avatar
Coach
Employee
mikewebguy
Employee

Likes

630 likes

Total Posts

402 posts

Correct Reply

85 solutions
Top badges earned
Coach
Contributor
Shape 25
Shape 1
Shape 10
View profile
mikewebguy
Employee

29-08-2019

nehak44235668​,

I have a simple rule: "Always trust Target" because Analytics can not count un-stitched data hits. See: Minimizing inflated visit and visitor counts in A4T Also it's really easy to accidentally inflate numbers in a custom analytics report when you think you are looking at xyz but really you are asking for abc. So if you got the "database data" from The Target team I'd go with that. However as you point out best to start clean.

If this satisfies your query please like, mark as helpful and mark as answer. Otherwise lets keep the conversation going. Hope you have a wonderful day.

Mihnea Docea | Technical Support Consultant | Customer Experience | Adobe | (:: 1 (800) 497-0335

Avatar

Avatar
Give Back 5
Level 3
skandg43264764
Level 3

Likes

31 likes

Total Posts

38 posts

Correct Reply

7 solutions
Top badges earned
Give Back 5
Give Back 3
Give Back 10
Give Back
Boost 5
View profile

Avatar
Give Back 5
Level 3
skandg43264764
Level 3

Likes

31 likes

Total Posts

38 posts

Correct Reply

7 solutions
Top badges earned
Give Back 5
Give Back 3
Give Back 10
Give Back
Boost 5
View profile
skandg43264764
Level 3

29-08-2019

You should trust Target platform to decide winner for an A4T activity, when you have confusion among Target and Analytics report.

Target report will provide you a global winner which you can take as statistical winner also.

Regards

Skand Gupt

Avatar

Avatar
Validate 1
Level 2
nehak44235668
Level 2

Likes

7 likes

Total Posts

9 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
View profile

Avatar
Validate 1
Level 2
nehak44235668
Level 2

Likes

7 likes

Total Posts

9 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
View profile
nehak44235668
Level 2

30-08-2019

So here's my problem with Target- The conversion for A and B based on visits is different from what I see in Analytics for our overall website conversion.

The directionality is different but also the conversion numbers. Target show a higher conversion compared to Analytics. The conversion in Analytics matches out overall website conversion. Also, the confidence level for the test is low like 76% and it's keeps changing drastically. It reached 98% confidence level in the first couple days but since then it's been very low. So we have two issues here- Directionality within platforms and confidence level. Even if I trust Target data, the confidence level is low. What is your recommendation for such situation. If we restart the test, and if we see B is winning but confidence level is still low we will stop the test. How much time should be given to such tests before you decide to quit based on confidence level?

I look forward to your thoughts.

Thank you.

Avatar

Avatar
Validate 1
Level 2
Sushmith_M_A_
Level 2

Likes

15 likes

Total Posts

7 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Shape 1
Boost 5
Boost 3
Boost 10
View profile

Avatar
Validate 1
Level 2
Sushmith_M_A_
Level 2

Likes

15 likes

Total Posts

7 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Shape 1
Boost 5
Boost 3
Boost 10
View profile
Sushmith_M_A_
Level 2

05-09-2019

Prior to activate any test you should decide how long should you run an A/B test.Simply monitoring test for long duration leads to false positive results( How long should you run an A/B test? ​).

Based on total number of Daily visitors you can calculate the test duration using below link.

https://docs.adobe.com/content/target-microsite/testcalculator.html

Regards,

Sushmitha

Avatar

Avatar
Give Back 50
Employee
Shruthi_N
Employee

Likes

272 likes

Total Posts

236 posts

Correct Reply

64 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile

Avatar
Give Back 50
Employee
Shruthi_N
Employee

Likes

272 likes

Total Posts

236 posts

Correct Reply

64 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 10
View profile
Shruthi_N
Employee

05-09-2019

Thank you SushmithaGowda​ for sharing that resource. Adding to what you said, Nine common A/B testing pitfalls and how to avoid them  is also a good read .