Control Group shows participation in Challenger Activities | Community
Skip to main content
M
mhallnorvell1
Level 2
September 28, 2023
Solved

Control Group shows participation in Challenger Activities

  • September 28, 2023
  • 1 reply
  • 582 views

We're running an A/B test, as the analyst monitoring the test, I'm seeing some users (one or two) appear to have been included in the Control Group but participated in an event activity that should only be visible in the Challenger group.  In this case, they clicked on a custom main nav link that was added only to the Challenger group. The test should be set up to exclude users in one test group from seeing the other.  What might be causing this to happen?  I did not set up the test but am an analyst.  Again, it's only 1 or 2, but if the users were in the Challenger group, why are they falling under the Control counts?

 

To add context there is personalization XT being applied to a subset of pages that could fall into the A/B test. But the challenger and control will see it equally. Could this be interfering with the A/B test?

 

Thanks,

 

Michelle

 

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by alexbishop

    It certainly sounds possible that the XT test could be interfering with the A/B test; it's best to avoid running multiple tests on the same page wherever possible. I think if those 1 or 2 users make up a very small % of overall traffic then it's going to be difficult to troubleshoot. However, if it's 1 or 2 users out of 20, then it would be worth getting someone to test the A Experience to make sure that there is no scenario where the custom nav link would be added

    1 reply

    alexbishop
    Adobe Employee
    alexbishopAdobe EmployeeAccepted solution
    Adobe Employee
    October 26, 2023

    It certainly sounds possible that the XT test could be interfering with the A/B test; it's best to avoid running multiple tests on the same page wherever possible. I think if those 1 or 2 users make up a very small % of overall traffic then it's going to be difficult to troubleshoot. However, if it's 1 or 2 users out of 20, then it would be worth getting someone to test the A Experience to make sure that there is no scenario where the custom nav link would be added

      Terms & ConditionsAccessibility statement

      Loading footer...