How do I read A4T confidence levels for my ab tests?

[RamizJa]

I asked this question in AA community but someone recommended that I post on here.

So I am trying to standardize the way my company is trying to ready ab test results. 

The main question I want to know is:

Do we read the overall significance OR the % given at the target experience level? 

66% or 93% ?

[alexbishop · Accepted solution]

You would read the individual experience level confidence % and then compare that against the confidence level that is set. By default this is 95% but you do have the option to export results data and adjust this (as mentioned here). It's also important to calculate the required sample size before starting the test, and checking that the "winning" Experience has met that sample size requirement, as part of validating the test results

[RamizJa]

So if I am operating at a 95% confidence level how would I interpret these results? 
I have two metrics that I care about and they both have seemed to reach 95+ confidence levels. 
The reason why I care about these two significances is because I want to both metrics in my calculations of a conversion rate = Total From Completes / Total Form Starts 

[alexbishop]

Typically you would just optimize against a single metric (i.e., Form Completions) and then look at other supporting metrics/segments/dimensions to get both a broader, but also more detailed, understanding of test performance. So you would ideally use that Primary KPI of Form Completions in the initial pre-test sample size calculations, conclude the test when the required sample size is met, and then evaluate the different metrics like confidence level etc

[RamizJa]

Thank you! That is helpful! I will keep the practice of following a single KPI for my sample size calculations and then keep with that to end the tests.

Follow-up question: 
How do you calculate the sample size using the Adobe calculator accurately? 
https://experienceleague.adobe.com/tools/calculator/testcalculator.html

-How do you decide on the baseline conversion rate and the total daily visitors?
-What is an example of a good page I should be using for the daily visits?
-Also for the conversion metrics baseline, what is a good way to get these metrics for our single KPI?

      normalizing for different behaviors because of months, campaigns, etc.

[alexbishop]

The Baseline Conversion Rate will be the current conversion rate (Metric/Visitors) for the Metric that you're optimising against. So if your test hypothesis is focused around increasing Form Starts, then you would take Form Starts/Visitors to find your baseline conversion rate. As you mention, you should take into consideration seasonality, ongoing promotions etc when deciding what lookback window/period to use to calculate this number

For the # of Daily Visitors, this should be specific to the page that your presenting the activity/offer on; this input is going to affect the number of days it takes to reach a statistically significant result, so it's important for the broader planning/prioritisation of the optimisation programme  

[RamizJa]

Thank you! This response was very helpful! If you have any more information or helpful links about the above topic feel free to share them here. 

[RamizJa]

Hi @alexbishop 
Using the recommendation you gave I calculated these numbers and it is saying I need to run my tests for 1 day for a 5% lift and 1 day for a 10% lift reading.

How do I correct interpret the information given from this calculator? 

[alexbishop]

Lift is also referred to as Minimum Detectable Effect (MDE); there are a lot of good articles that talk about this in detail but effectively it reflects the change in conversion rate you expect from your variant; the required sample size becomes smaller as the MDE becomes bigger.

As part of the calculations you are assuming an MDE; however, you would then need to review this, along with the statistical significance, when you are considering concluding the test e.g. you assumed an MDE of 10% and therefore a sample size of X but in reality saw an MDE of 5%, in which case the required sample size would be higher than X

[AnubhavRa1]

What does the overall confidence% mean? Is it worth interpreting in any way?

thanks for your help!

[alexbishop]

The main focus should be on the individual experience-level calculations vs the Control