How do I read A4T confidence levels for my ab tests? | Community
Skip to main content
R
RamizJa
Level 2
February 14, 2024
Solved

How do I read A4T confidence levels for my ab tests?

  • February 14, 2024
  • 1 reply
  • 2588 views

I asked this question in AA community but someone recommended that I post on here.

So I am trying to standardize the way my company is trying to ready ab test results. 

 

The main question I want to know is:

 

Do we read the overall significance OR the % given at the target experience level? 

 

66% or 93% ?

 

 

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by alexbishop

    You would read the individual experience level confidence % and then compare that against the confidence level that is set. By default this is 95% but you do have the option to export results data and adjust this (as mentioned here). It's also important to calculate the required sample size before starting the test, and checking that the "winning" Experience has met that sample size requirement, as part of validating the test results

    1 reply

    alexbishop
    Adobe Employee
    alexbishopAdobe EmployeeAccepted solution
    Adobe Employee
    February 15, 2024

    You would read the individual experience level confidence % and then compare that against the confidence level that is set. By default this is 95% but you do have the option to export results data and adjust this (as mentioned here). It's also important to calculate the required sample size before starting the test, and checking that the "winning" Experience has met that sample size requirement, as part of validating the test results

      R
      RamizJaAuthor
      Level 2
      February 15, 2024

      So if I am operating at a 95% confidence level how would I interpret these results? 
      I have two metrics that I care about and they both have seemed to reach 95+ confidence levels. 
      The reason why I care about these two significances is because I want to both metrics in my calculations of a conversion rate = Total From Completes / Total Form Starts 

       

        alexbishop
        Adobe Employee
        alexbishopAdobe Employee
        Adobe Employee
        February 16, 2024

        Typically you would just optimize against a single metric (i.e., Form Completions) and then look at other supporting metrics/segments/dimensions to get both a broader, but also more detailed, understanding of test performance. So you would ideally use that Primary KPI of Form Completions in the initial pre-test sample size calculations, conclude the test when the required sample size is met, and then evaluate the different metrics like confidence level etc

          Terms & ConditionsAccessibility statement

          Loading footer...