Acceptable Traffic Split Deviation in Adobe Target A/B Testing

[ShoebSh]

I am running an A/B test in Adobe Target with a 50/50 traffic split configuration. However, upon reviewing the data, I noticed that the actual split is 49.8% (variant) vs. 50.5% (control)—a difference of approximately 0.7%.

I would like to understand:

Is this level of deviation (~0.7%) considered normal in Adobe Target A/B tests?
Does Adobe Target enforce an exact 50/50 split, or is there an expected margin of error?
At what level of imbalance should I be concerned about test validity?
The test is still ongoing, and I want to ensure that this traffic allocation discrepancy does not impact the final results. Any official Adobe documentation or expert insights would be greatly appreciated.

[Vaibhav_Mathur · Accepted solution]

Hi @ShoebSh , 

Quoting few lines from  the documentation, "As a result of random splitting of traffic, it can take a significant amount of initial traffic before the percentages even out. For example, if you create two experiences, the starting experience is chosen randomly. If there is little traffic, it’s possible that the percentage of visitors can be skewed toward one experience. As traffic increases, the percentages equalize."

Reference : https://experienceleague.adobe.com/en/docs/target/using/introduction/how-target-works 

I don't think there is any given acceptable level of deviation in the documents, but a normal expected deviation according to my experience in less traffic pages can be around 1%-2%.

A best practice to manage this deviation can be to run an A/A test on the experiment page beforehand to analyse the traffic and the accuracy of traffic allocation on the given page. 

For reference : https://experienceleague.adobe.com/en/docs/target/using/activities/abtest/aa-testing 

I hope it helps. 

Best regards, 

Vaibhav Mathur

[ShoebSh]

Hi @Vaibhav_Mathur Thank you for your detailed response and guidance.

[ShariLynnDeutsch]

Hi @ShoebSh ,

@Vaibhav_Mathur  is correct. The discrepancy is called sample ratio mismatch.  I use the Speero calculator or an internal one to determine if it has compromised my experiment. The side of the audience will impact the allowable amount of discrepancy. While I keep an eye on the split during an experiment, we can only get a true SRM reading once the experiment is complete. (one side may end up with more temporarily, as shown in the documentation)   In the end, we usually end up with < 1% difference between the two experiences.

More info than you asked for, but I thought that this might help.

Thanks!

Shari

@Vai

[ShoebSh]

Hi @ShariLynnDeutsch. Thank you for your response.