So I am trying to standardize the way my company is trying to ready ab test results.
The main question I want to know is:
Do we read the overall significance OR the % given at the target experience level?
66% or 93% ?
Solved! Go to Solution.
Views
Replies
Total Likes
Confidence tells you which version won. The confidence Level indicates the likelihood that the results would be duplicated if the test were run again. Say you ran a test with offer A and offer B in which Offer B won with a Confidence level of 95%. That means if you ran the same exact test again, there is a 95% chance that offer B would win the second time (and a 5% chance it would not). For this reason, you typically look for a confidence level between 95% and 99% before declaring a winner.
Lift tells you by how much a version won. In the Target report for Workspace there are three Lift metrics: Lift (Lower), Lift (Mid), and Lift (Upper). Lift compares the conversion rate for each experience against the control experience. Lift (Lower) represents the worst lift a variant experience could have over the control, at a 95% confidence interval. Lift (Mid) represents the midpoint lift a variant experience could have over the control, at a 95% confidence interval. Lift (Upper) represents the best lift a variant experience could have over the control, at a 95% confidence interval. The spread between the lower and upper lift numbers represents the confidence in the accuracy of the estimated conversion rates. The larger the sample size (more visitors) the more confident you can be that the estimates of the conversion rates are accurate. To keep things simple, I used the Mid number.
This might be worth cross posting in the Target Forum... I don't have Target, so I am not sure the appropriate answer here.
https://experienceleaguecommunities.adobe.com/t5/adobe-target/ct-p/adobe-target-community
^ Someone there may be able to help you better
Good Luck.
Views
Replies
Total Likes
Confidence tells you which version won. The confidence Level indicates the likelihood that the results would be duplicated if the test were run again. Say you ran a test with offer A and offer B in which Offer B won with a Confidence level of 95%. That means if you ran the same exact test again, there is a 95% chance that offer B would win the second time (and a 5% chance it would not). For this reason, you typically look for a confidence level between 95% and 99% before declaring a winner.
Lift tells you by how much a version won. In the Target report for Workspace there are three Lift metrics: Lift (Lower), Lift (Mid), and Lift (Upper). Lift compares the conversion rate for each experience against the control experience. Lift (Lower) represents the worst lift a variant experience could have over the control, at a 95% confidence interval. Lift (Mid) represents the midpoint lift a variant experience could have over the control, at a 95% confidence interval. Lift (Upper) represents the best lift a variant experience could have over the control, at a 95% confidence interval. The spread between the lower and upper lift numbers represents the confidence in the accuracy of the estimated conversion rates. The larger the sample size (more visitors) the more confident you can be that the estimates of the conversion rates are accurate. To keep things simple, I used the Mid number.
Thank you Robert! This is very helpful. I still need clarification on whether to use the overall confidence give on an A4T panel in AA or at the Adobe Target level. See the picture below:
Overall - 66.38% Confidence
Travel Protection Recommendation - 93.79% Confidence
Views
Replies
Total Likes
Views
Likes
Replies