Sample Size Calculator Specifics

[Adobeuser456]

We're running an A/B/C test and trying to determine how to use the sample size calculator properly. We will stick with the recommended 95% confidence level and 80% statistical power, and we have 3 offers (including control). But little confused on baseline conversion rate and daily visitors.

Our test is showing 2 different product groupings on a specific page (vs. the control). Ultimately, we're trying to drive as much revenue (this is our success metric; normalizing metric in A4T is unique visitors) as possible, so we're running the test to determine which product grouping drives the highest revenue.

So in this case, should total number of daily visitors be...

• average daily unique visitors to that specific page where the test is being run?
• average daily unique visitors to the site overall?
• total visits to that specific page?
• something else?

And should baseline conversion rate just be whatever our normal conversion rate (orders/visits) for the overall site? Or something else or more specific?

[Gokul_Agiwal · Accepted solution]

Hi @Adobeuser456 

For your scenarios - 

For the baseline conversion rate, you should use the existing conversion rate for the specific page where the test is being run, not the overall site conversion rate.  The conversion rate should be the order/visit ratio for that specific page, not the entire site.
This will provide a more accurate representation of the starting point for your test.

About Daily Visitors:

The correct metric to use for the "Daily Visitors" in the sample size calculator is the average daily unique visitors to the specific page where the test is being run.  This should not be the overall site traffic, as the test is focused on a specific page.

And as you mentioned you'll stick to the recommended 95% confidence level and 80% statistical power as this are the industry standard anyway. 

Thanks

[pradnya_balvir]

Hi @Adobeuser456 ,

I agree with @Gokul_Agiwal . 

For conversion : Order/visits for that specific page.

[Adobeuser456]

Hello @pradnya_balvir and @Gokul_Agiwal - since this page itself is not transactable...I imagine you mean orders that included a visit to that page divided by overall visits to that page?