How do I read A4T confidence levels for my ab tests? | Adobe Higher Education
Skip to main content
R
RamizJa
Level 2
February 14, 2024
解決済み

How do I read A4T confidence levels for my ab tests?

  • February 14, 2024
  • 1 の返信
  • 2588 ビュー

I asked this question in AA community but someone recommended that I post on here.

So I am trying to standardize the way my company is trying to ready ab test results. 

 

The main question I want to know is:

 

Do we read the overall significance OR the % given at the target experience level? 

 

66% or 93% ?

 

 

    このトピックへの返信は締め切られました。
    ベストアンサー alexbishop

    You would read the individual experience level confidence % and then compare that against the confidence level that is set. By default this is 95% but you do have the option to export results data and adjust this (as mentioned here). It's also important to calculate the required sample size before starting the test, and checking that the "winning" Experience has met that sample size requirement, as part of validating the test results

    1 の返信

    alexbishop
    Adobe Employee
    alexbishopAdobe Employee回答
    Adobe Employee
    February 15, 2024

    You would read the individual experience level confidence % and then compare that against the confidence level that is set. By default this is 95% but you do have the option to export results data and adjust this (as mentioned here). It's also important to calculate the required sample size before starting the test, and checking that the "winning" Experience has met that sample size requirement, as part of validating the test results

      R
      RamizJa作成者
      Level 2
      February 15, 2024

      So if I am operating at a 95% confidence level how would I interpret these results? 
      I have two metrics that I care about and they both have seemed to reach 95+ confidence levels. 
      The reason why I care about these two significances is because I want to both metrics in my calculations of a conversion rate = Total From Completes / Total Form Starts 

       

        R
        RamizJa作成者
        Level 2
        February 21, 2024

        The Baseline Conversion Rate will be the current conversion rate (Metric/Visitors) for the Metric that you're optimising against. So if your test hypothesis is focused around increasing Form Starts, then you would take Form Starts/Visitors to find your baseline conversion rate. As you mention, you should take into consideration seasonality, ongoing promotions etc when deciding what lookback window/period to use to calculate this number

        For the # of Daily Visitors, this should be specific to the page that your presenting the activity/offer on; this input is going to affect the number of days it takes to reach a statistically significant result, so it's important for the broader planning/prioritisation of the optimisation programme  


        Hi @alexbishop 
        Using the recommendation you gave I calculated these numbers and it is saying I need to run my tests for 1 day for a 5% lift and 1 day for a 10% lift reading.

        How do I correct interpret the information given from this calculator? 

         

         

        契約条件Accessibility statement

        Loading footer...