Expand my Community achievements bar.

SOLVED

How do I read A4T confidence levels for my ab tests??

Avatar

Level 2

So I am trying to standardize the way my company is trying to ready ab test results. 

 

The main question I want to know is:

 

Do we read the overall significance OR the % given at the target experience level? 

 

66% or 93% ?

1 Accepted Solution

Avatar

Correct answer by
Level 10

Confidence tells you which version won. The confidence Level indicates the likelihood that the results would be duplicated if the test were run again. Say you ran a test with offer A and offer B in which Offer B won with a Confidence level of 95%. That means if you ran the same exact test again, there is a 95% chance that offer B would win the second time (and a 5% chance it would not). For this reason, you typically look for a confidence level between 95% and 99% before declaring a winner.

Lift tells you by how much a version won. In the Target report for Workspace there are three Lift metrics: Lift (Lower), Lift (Mid), and Lift (Upper). Lift compares the conversion rate for each experience against the control experience. Lift (Lower) represents the worst lift a variant experience could have over the control, at a 95% confidence interval. Lift (Mid) represents the midpoint lift a variant experience could have over the control, at a 95% confidence interval. Lift (Upper) represents the best lift a variant experience could have over the control, at a 95% confidence interval. The spread between the lower and upper lift numbers represents the confidence in the accuracy of the estimated conversion rates. The larger the sample size (more visitors) the more confident you can be that the estimates of the conversion rates are accurate. To keep things simple, I used the Mid number.

View solution in original post

3 Replies

Avatar

Community Advisor and Adobe Champion

This might be worth cross posting in the Target Forum... I don't have Target, so I am not sure the appropriate answer here.

 

https://experienceleaguecommunities.adobe.com/t5/adobe-target/ct-p/adobe-target-community

 

^ Someone there may be able to help you better

 

Good Luck.

Avatar

Correct answer by
Level 10

Confidence tells you which version won. The confidence Level indicates the likelihood that the results would be duplicated if the test were run again. Say you ran a test with offer A and offer B in which Offer B won with a Confidence level of 95%. That means if you ran the same exact test again, there is a 95% chance that offer B would win the second time (and a 5% chance it would not). For this reason, you typically look for a confidence level between 95% and 99% before declaring a winner.

Lift tells you by how much a version won. In the Target report for Workspace there are three Lift metrics: Lift (Lower), Lift (Mid), and Lift (Upper). Lift compares the conversion rate for each experience against the control experience. Lift (Lower) represents the worst lift a variant experience could have over the control, at a 95% confidence interval. Lift (Mid) represents the midpoint lift a variant experience could have over the control, at a 95% confidence interval. Lift (Upper) represents the best lift a variant experience could have over the control, at a 95% confidence interval. The spread between the lower and upper lift numbers represents the confidence in the accuracy of the estimated conversion rates. The larger the sample size (more visitors) the more confident you can be that the estimates of the conversion rates are accurate. To keep things simple, I used the Mid number.

Avatar

Level 2

Thank you Robert! This is very helpful. I still need clarification on whether to use the overall confidence give on an A4T panel in AA or at the Adobe Target level. See the picture below:
Overall - 66.38% Confidence
Travel Protection Recommendation - 93.79% Confidence 

RamizJa_0-1707955526808.png