How do I read A4T confidence levels for my ab tests?? | Community
Skip to main content
Level 2
February 14, 2024
Solved

How do I read A4T confidence levels for my ab tests??

  • February 14, 2024
  • 2 replies
  • 2018 views

So I am trying to standardize the way my company is trying to ready ab test results. 

 

The main question I want to know is:

 

Do we read the overall significance OR the % given at the target experience level? 

 

66% or 93% ?

Best answer by RobertBlakeley

Confidence tells you which version won. The confidence Level indicates the likelihood that the results would be duplicated if the test were run again. Say you ran a test with offer A and offer B in which Offer B won with a Confidence level of 95%. That means if you ran the same exact test again, there is a 95% chance that offer B would win the second time (and a 5% chance it would not). For this reason, you typically look for a confidence level between 95% and 99% before declaring a winner.

Lift tells you by how much a version won. In the Target report for Workspace there are three Lift metrics: Lift (Lower), Lift (Mid), and Lift (Upper). Lift compares the conversion rate for each experience against the control experience. Lift (Lower) represents the worst lift a variant experience could have over the control, at a 95% confidence interval. Lift (Mid) represents the midpoint lift a variant experience could have over the control, at a 95% confidence interval. Lift (Upper) represents the best lift a variant experience could have over the control, at a 95% confidence interval. The spread between the lower and upper lift numbers represents the confidence in the accuracy of the estimated conversion rates. The larger the sample size (more visitors) the more confident you can be that the estimates of the conversion rates are accurate. To keep things simple, I used the Mid number.

2 replies

Jennifer_Dungan
Community Advisor and Adobe Champion
Community Advisor and Adobe Champion
February 14, 2024

This might be worth cross posting in the Target Forum... I don't have Target, so I am not sure the appropriate answer here.

 

https://experienceleaguecommunities.adobe.com/t5/adobe-target/ct-p/adobe-target-community

 

^ Someone there may be able to help you better

 

Good Luck.

RobertBlakeley
Community Advisor
RobertBlakeleyCommunity AdvisorAccepted solution
Community Advisor
February 14, 2024

Confidence tells you which version won. The confidence Level indicates the likelihood that the results would be duplicated if the test were run again. Say you ran a test with offer A and offer B in which Offer B won with a Confidence level of 95%. That means if you ran the same exact test again, there is a 95% chance that offer B would win the second time (and a 5% chance it would not). For this reason, you typically look for a confidence level between 95% and 99% before declaring a winner.

Lift tells you by how much a version won. In the Target report for Workspace there are three Lift metrics: Lift (Lower), Lift (Mid), and Lift (Upper). Lift compares the conversion rate for each experience against the control experience. Lift (Lower) represents the worst lift a variant experience could have over the control, at a 95% confidence interval. Lift (Mid) represents the midpoint lift a variant experience could have over the control, at a 95% confidence interval. Lift (Upper) represents the best lift a variant experience could have over the control, at a 95% confidence interval. The spread between the lower and upper lift numbers represents the confidence in the accuracy of the estimated conversion rates. The larger the sample size (more visitors) the more confident you can be that the estimates of the conversion rates are accurate. To keep things simple, I used the Mid number.

RamizJaAuthor
Level 2
February 15, 2024

Thank you Robert! This is very helpful. I still need clarification on whether to use the overall confidence give on an A4T panel in AA or at the Adobe Target level. See the picture below:
Overall - 66.38% Confidence
Travel Protection Recommendation - 93.79% Confidence 

 

TimBo4
Level 2
September 25, 2025

This is good question. Does the overall confidence have any significance? @robertblakeley