Control Group shows participation in Challenger Activities

[mhallnorvell1]

We're running an A/B test, as the analyst monitoring the test, I'm seeing some users (one or two) appear to have been included in the Control Group but participated in an event activity that should only be visible in the Challenger group.  In this case, they clicked on a custom main nav link that was added only to the Challenger group. The test should be set up to exclude users in one test group from seeing the other.  What might be causing this to happen?  I did not set up the test but am an analyst.  Again, it's only 1 or 2, but if the users were in the Challenger group, why are they falling under the Control counts?

To add context there is personalization XT being applied to a subset of pages that could fall into the A/B test. But the challenger and control will see it equally. Could this be interfering with the A/B test?

Thanks,

Michelle

[alexbishop · Accepted solution]

It certainly sounds possible that the XT test could be interfering with the A/B test; it's best to avoid running multiple tests on the same page wherever possible. I think if those 1 or 2 users make up a very small % of overall traffic then it's going to be difficult to troubleshoot. However, if it's 1 or 2 users out of 20, then it would be worth getting someone to test the A Experience to make sure that there is no scenario where the custom nav link would be added