Clarification on Lift & Confidence in Analytics Dashboard for Personalization

[AnishKo]

I have created four A/B test activities for personalization. Each activity has the same control experience with 5% traffic allocation and a test experience with 95% traffic allocation. All four activities target different audiences but run on the same page URL.

When reviewing the Analytics for Target table in the analytics dashboard, I’m seeing large variations in lift and confidence. In this scenario, what is the correct way to calculate lift and confidence?

[bjoern__koth]

Hi @AnishKo 

TL;DR; you cannot combine or mentally “average” controls across activities, and doing so is statistically invalid. Try increasing the control size (why not go with a 50/50 split per experience?) and your data should at least gain stastistical significance.

A 5% control group size is very small and is statistically fragile, especially when audiences differ in size, behavior, or baseline conversion. Your confidence fluctuates because the control sample is too small to stabilize variance.

Instead, each activity’s lift and confidence are valid only for that audience and that activity.

With your 5% control size, results are directionally indicative at best, but not decision-grade.

Hence, the variation is expected and is a consequence of underpowered controls, not an analytics error.