Manual Allocation Set At 50:50, Actually Splitting 60:40 (and other issues)

[jordanb48838092]

Hey,

We've recently implemented Target at my work having moved away from Optimizely, so apologies if these issues seem very basic. I have looked on the forums but to no avail.

We've had a fair few issues with A/B tests, so far we've only used manual allocation so the issues pertain to that. Hopefully our problems are connected and there's one cause we're not quite seeing.

Firstly, as the title suggests the splits are not matching what we set the allocation at when we look at unique visitors (we're using Adobe Analytics). 50:50 tests have tended to go 60:40, and a 60:20:20 test went 70:15:15.

Another issue we're seeing is a test that ran with no difference between A and B saw an uplift of roughly 8% at 99.99% interval - which obviously seems very unlikely.

There are a few other small issues that mean we currently can't really trust our results but those are the main ones.

As I say we're fairly new to Target so any help/possible suggestions would be greatly appreciated - we haven't had much luck emailing our contacts at Adobe.

Thank you,

Jordan

[archits30249934]

Hi Jordan,

There could be multiple factors affecting the traffic split among your control and challenger variants.

Firstly, total traffic in the test. If the test is in the initial stage after launch and the numbers are small, you might see an unequal split. However, as the traffic increases the split should be as you have set manually in Target. Example, for a low traffic page(100 Visitors per page) you might not see an equal split initially.

Secondly, Target and Analytics have different engines to count visitors. Yes, if you are using Analytics for Target (aka A4T) numbers should be equal and there will be an equal split. If you are using a custom code(sending Control or Challenger information into a variable) to send traffic to Adobe Analytics, you might need to check if the code runs every time.

You might also want to check the advanced settings for the metrics set up in Target(Success Metrics).

Hope this is helpful.

Thanks,

Archit

[tanvia28417128]

HI jordanb48838092

Adding to Archit's comment, As Target and Analytics have different engines to count visitors, it is recommended to verify Traffic split of A4T activities through Target activity performance APIs. APIs can be referred here >  Target API v1.0

Hope that helps.

[jordanb48838092]

Thank you sainiarchit​ and tanvia21455990​ for your help.

Just to answer some questions, the visitor numbers should be large enough that the uneven split can't be waved away - in the hundreds of thousands.

We are using and seeing the discrepancy in Analytics for Target numbers, not custom code.

From what I gather we don't have access to the advanced settings for the metrics in Target as we're using A4T.

I'm having trouble getting the API link to work, but can probably figure that out and will look into it.

If you think of anything else that would be helpful please let me know

Thank you,

Jordan