Running an A/A Test: Seeing difference in performance between 2 controls

[jojo22323]

Hello,

I am running an A/A test and started including a second control in all my tests so I can do an A/A comparison (10% of traffic to second control) in every test.  The issue is that the two controls perform differently a lot of the time, even with a even 50/50 split.

Questions:

• At what point do you consider two controls to be performing similarly enough to trust the baseline?  Is it =-3% conversion rate for the KPI?  It is hard to do anything higher if your power analysis has a minimum detectable effect of 5%.
• How big of a sample is needed for Target to determine two controls have the same performance?  What can impact that time?  Wondering if I do a typical pre-test duration calculation/power analysis and run the tests longer if the controls are not performing the same.

The biggest issue with the A/A test results is that it puts in question the previous tests we have run as we have no idea if the control performance was actually a true representation of the baseline.

Thank you for any help!

[niocolasmeriel · Accepted solution]

Hi Jojo

you might want to check out this ressource (if not already done):

https://experienceleague.adobe.com/docs/target/using/activities/abtest/sample-size-determination.htm...

Not sure how much traffic/conversion and runtime you had in your setup but I would recommend you consider at least 2-4 weeks of runtime if not more (depending on traffic/conversion too).

Cheers

Nicolas