Target's traffic allocation is not 50:50 with redirect tests

[floriana_guardi]

We have run a few redirection A/B tests, the B version is created in AEM and 50% of the users should be redirected to it. Redirection is usually used when there is a major change on the page which cannot be done either with code change or using fragments.

What we observe in all instances is that, as opposed to tests where we inject a change on the page, the traffic allocation is never 50:50 and the difference is large enough to result in a Sample Ratio Mismatch. This is very problematic as it impacts the trustworthiness of our test results and puts the allocation algorithm from Adobe in question.

This issue seems to impact mobile traffic at a much higher rate than desktop traffic.

My question is:

What is creating this issue?

How do we contain it if we want to run redirection tests like this in the future?

Thank you for your input!

[Gaurav_Singh · Accepted solution]

@floriana_guardi Could you please specify if we are using Target or Analytics as a reporting source ?

With Target as a reporting source , you should not be facing issues as the reporting is very straight forward. 

The problem may come up when we are using Analytics as a reporting source (A4T) because then there could be scenario's wherein SDID parameter for Target call and Analytics call are not matching which may lead to unstitched hits which may further lead to incorrect traffic distribution.

Few key pointers to keep in mind while creating a A4T redirect activity :

1. Library versions for Target , Analytics and Marketing cloud visitor ID.

2. Firing order of these libraries (Marketing cloud visitor ID > Target > Analytics).

3. Use only default redirect offer present in Target. Custom redirect offers are not supported.

4. Ensure SDID for Target and Analytics call are same.

If you are still facing this issue with redirect offers , I would recommend to initiate a support ticket so we can evaluate the setup and suggest accordingly.

[floriana_guardi]

Hi @Gaurav_Singh , the discrepancy is visible both in Target report and Analytics with A4T. I'm not involved in the technical setup, so I can't comment on the pointers, but I will check with my developer. I have also raised a support ticket, thank you for the suggestion!