Adobe Target - 50/50 traffic split - test two experience vs their own controls

Avatar

Avatar

ryanj16539978

Avatar

ryanj16539978

ryanj16539978

25-07-2018

My team is stuck on this being an A/B test rather than if the control performs better than a customized experience. My original plan was to send 50% of the traffic to the control and 25% to Exp B and 25% to Exp C. The results of B and C would be combined because the primary goal of the test is to justify further testing in personalization/targeting. Is there a better way to look at this? I had found some documentation within Adobes resources but I can't find it to save my life now.

Thanks

View Entire Topic

Avatar

Avatar

LJ_Jones

Avatar

LJ_Jones

LJ_Jones

25-07-2018

Can you give a little more info?  What do you mean, "they are stuck on this being an A/B"?  What is the push back you are getting?  What are their concerns? Is there an internal debate on the validity of personalization?

A couple thoughts here.

  1. Define your goal.  It doesn't sound like everyone is on the same page as far as what they goal for this test is.  Its key to get everyone on the same page with the same goal before testing.  The goal should define the test.
  2. Personalization can be tricky.  It may be difficult with a single test to prove that its worth it to justify more personalization testing.  Imagine if this test fails.  Does that mean you won't try other personalization efforts?  Personalization is a the right way to go, but it may take multiple tests and trial and error to find the right personalizations that work.
  3. I like your setup.  I think this is a good setup because you can look at B and C both individually and collectively.  As I said in point 2, it can be tricky.  You may find that B isn't a good personalization, but but your customers do respond to C.  You may even consider going 33% traffic for each experience.  You can't reach significance until each experience has had enough traffic. 
  4. I would be more concerned with finding personalizations that work for your customers than proving that personalization itself is a good strategy.