Calculating significance/test duration for a null hypothesis | Community
Skip to main content
T
tcrewes1
Level 3
May 21, 2024
Solved

Calculating significance/test duration for a null hypothesis

  • May 21, 2024
  • 1 reply
  • 758 views

Hi everyone,
I have a stakeholder who wants to run an AB test which involves removing a panel from a page on our site. The panel is near the bottom of the page, and it has a few UX issues.
Their hypothesis is that removing this panel will not have a significant impact on users finding the content featured on it, as users will find the content through other navigational means.
I'm trying to establish how we can be confident that the removal has had no impact. When looking at the Adobe Statistical Significance calculator, I get an "Infinity" in the number of days when I put "0%" in the expected uplift field.

Has anyone had this issue before/come up with a way of establishing significance for a null hypothesis test. 

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by kandersen-1

    I don't have much experience with null hypothesis, but here's my 2 cents - if it is even worth that much 🙂

     

    I believe that with a null hypothesis test where you expect no change or impact (0% uplift), traditional significance calculators (like the adobe target one) might struggle, as they are designed to detect differences rather than confirm the absence of a difference. At least that is my understanding.

     

    Well, aware I'm not answering your question, I would approach it slightly different.

     

    Assuming you have Adobe Analytics i would try and get insights that confirms or debunks the hypothesis - if you don't have specific tracking (e.g. clicks) on the content you may be able to find number of clicks via ActivityMap dimensions (assuming you have this enabled) to give you an idea of how much traffic/clicks this module is actually driving to the content it features. 

     

    The analytics implementation can also easily be enhanced to provide these insights by simply setting an eVar with the value of the navigation method (It is a tracking fundamental typically named ' product finding methods'). e.g. when someone performs a search you populate the evar with 'internal search', when they use menu navigation you populate (the same evar) with 'navigation', etc. etc. 

     

    I hope you figure it out - if so, please post your solution, as I'd be interested in learning more about this.

    1 reply

    kandersen-1
    Community Advisor
    kandersen-1Community AdvisorAccepted solution
    Community Advisor
    May 23, 2024

    I don't have much experience with null hypothesis, but here's my 2 cents - if it is even worth that much 🙂

     

    I believe that with a null hypothesis test where you expect no change or impact (0% uplift), traditional significance calculators (like the adobe target one) might struggle, as they are designed to detect differences rather than confirm the absence of a difference. At least that is my understanding.

     

    Well, aware I'm not answering your question, I would approach it slightly different.

     

    Assuming you have Adobe Analytics i would try and get insights that confirms or debunks the hypothesis - if you don't have specific tracking (e.g. clicks) on the content you may be able to find number of clicks via ActivityMap dimensions (assuming you have this enabled) to give you an idea of how much traffic/clicks this module is actually driving to the content it features. 

     

    The analytics implementation can also easily be enhanced to provide these insights by simply setting an eVar with the value of the navigation method (It is a tracking fundamental typically named ' product finding methods'). e.g. when someone performs a search you populate the evar with 'internal search', when they use menu navigation you populate (the same evar) with 'navigation', etc. etc. 

     

    I hope you figure it out - if so, please post your solution, as I'd be interested in learning more about this.

    Test forum signature
      Terms & ConditionsAccessibility statement

      Loading footer...