Adobe Target - 50/50 traffic split - test two experience vs their own controls

[ryanj16539978]

My team is stuck on this being an A/B test rather than if the control performs better than a customized experience. My original plan was to send 50% of the traffic to the control and 25% to Exp B and 25% to Exp C. The results of B and C would be combined because the primary goal of the test is to justify further testing in personalization/targeting. Is there a better way to look at this? I had found some documentation within Adobes resources but I can't find it to save my life now.

Thanks

[Shruthi_N]

Hi, Maybe you are referring to the multipage activity documentation ( Multipage Activity ) . Hope that helps.

[ryanj16539978]

I don't think that is quite right. The test would need to have 1 or 2 controls, with one experience designed specifically for local traffic and another designed specifically for tourism traffic. The audiences would be driven by GEO.

[LJ_Jones]

Can you give a little more info?  What do you mean, "they are stuck on this being an A/B"?  What is the push back you are getting?  What are their concerns? Is there an internal debate on the validity of personalization?

A couple thoughts here.

1. Define your goal.  It doesn't sound like everyone is on the same page as far as what they goal for this test is.  Its key to get everyone on the same page with the same goal before testing.  The goal should define the test.
2. Personalization can be tricky.  It may be difficult with a single test to prove that its worth it to justify more personalization testing.  Imagine if this test fails.  Does that mean you won't try other personalization efforts?  Personalization is a the right way to go, but it may take multiple tests and trial and error to find the right personalizations that work.
3. I like your setup.  I think this is a good setup because you can look at B and C both individually and collectively.  As I said in point 2, it can be tricky.  You may find that B isn't a good personalization, but but your customers do respond to C.  You may even consider going 33% traffic for each experience.  You can't reach significance until each experience has had enough traffic. 
4. I would be more concerned with finding personalizations that work for your customers than proving that personalization itself is a good strategy. 

[kimenw]

Check out multiple experience versions - it's intended exactly for this use case! Multiple Experience Versions in an A/B Test

[ryanj16539978]

That's it! Apparently, I just needed lunch as I had been staring at that page before and couldn't remember the context. Thank you. So one experience will be local(50 miles) traffic and the other will be the rest of the nation. Since my two experiences are provided different content based on which they fall into I would need to use Auto-Target for allocation correct?

[ryanj16539978]

I think I might still be misunderstanding this setup just a bit still. If my three variations of content are 1. Control 2. Local Audience 3. Tourist Audience how do I maintain 50% of traffic towards control and then split the remaining based on geo experience?

[LJ_Jones]

You basically have two options to set this up.

1.  You can set this up using the visual experience composer.  It will be structured a little different than what you are describing.  Basically you will have two experiences with two audiences.  Since its a test, Target wants to test each experience against the same audience.  So its not possible to set it up as an experience with an audience vs a different experience with a different audience.  It wouldn't be a fair apples to apples comparison.  It would look like this.

Control:

• Local Audience
• Tourist Audience

Exp B

• Local Audience
• Tourist Audience

Here is a screen of how you get to setting up multiple audiences.

2.  The other option, is to set up three different tests.  So three tests each with one experience and one audience.  The problem here is that Target won't be able to compare the conversion rates for you.  You would have to do the comparison yourself using a spreadsheet, which I often do anyway.

Does this make sense?

[ryanj16539978]

So with the first scenario, the Exp A/Control would not have any personalized content on it but targets to both audiences and the personalized content would be present on Exp B. Correct?

For your second option, I can't run three tests on the same page at the same time correct?

[LJ_Jones]

On the first scenario, you are correct.  This is probably your best option.

For the second, it gets a little complicated.  You could run multiple tests on the same page but to different audiences.  The audience keeps them from being in both tests.  So you would have to take care to make sure that your audiences don't overlap.  It can get a little messy if you're not careful.  But it can also be effective for certain situations.

[ryanj16539978]

So my Exp A/Control would deliver identical content to both tourist and local traffic. But will my Exp B deliver traffic based on the Geo targeting placed into the two different audiences? If so, which traffic allocation method would make sure it chose the correct experience based on geo defined audiences? My guess is auto-target but I'm not 100% convinced.

Otherwise, doing two separate concurrent a/b tests with very specific geo requirements for each campaign will probably end up being the best case scenario.

[LJ_Jones · Accepted solution]

Yes, you are correct.  Exp B will basically have two experiences in it.  One for each audience.  Within visual experience composer, you can select the audience and then create an experience specifically for them.  Go into Audiences first and set up your two audiences based on GEO.   Then add those to the test.

I don't think I would use auto-allocate.  That just starts to show the new visitors the better performing exp.  I would just leave it at manual 50/50.  What that will do is make it so that 50% of your tourist audience sees the control and 50% sees Exp B for tourists.  And also 50% of the local audience will see control and 50% will see Exp B for local.