Help required with efficiently calculating statistical signifiance in adobe workspace

[Venky235]

Our team does A/B testing rigorously. I am from reporting side and for the initial reporting for every A/B test, we have a adobe dashboard explaining the results with KPIs like impressions, clicks, CTR, lift and few down stream metrics for each version. Typically, we have 2-4 versions for very test - one control and 3 variants. I want to add statistical significance in each dashboard. I have calculated it using the below logic.

Imagine a test with only 2 recipes - 1 control and 1 variant

CTR 1 = clicks1/impressions1

CTR2 = clicks 2/impressions 2

Pooled CTR = (clicks 1+clicks 2)/(impressions 1+impressions 2)

Standard Error (SE) = Square root(Pooled CTR*(1-Pooled CTR)*((1/Imp1)+(1/Imp2)))

Z Score = abs(CTR1-CTR2)/SE

Standard significance = cdf-z function (Z score)

I create every calculated metric separately and finally calculate the standard significance. But this is not scalable. Because I have to create 7 calculated metrics for each test. So if a A/B test has 4 variants, I would be needing statistical significance to be calculated 3 times against the control (Which makes it 7*3=21 calculated metrics).

Is there any easy way to do this? Any efficient way of achieving this would be of great help. Thanks!

[Nick_Walter · Accepted solution]

Unless you are reusing the same naming for different test variables, then I do not think that there is a scalable way to do these tests in workspace. If I were you, I would consider setting up an Excel report builder report that flows into a table and automate in Excel for this. This adds reporting outside of workspace which isn't great but you can add a link into a text box in workspace to easily link to the external report.

[Jennifer_Dungan]

Can you not make your clicks and impressions shared for all tests, and then just create a segment for each test that can be applied to the panel?

I'm not sure what A/B tool you are using, but when I implemented ours, I actually used s.products and merchandising eVars to capture our data...

So I have some standard metrics that are used for each variation's impression, and a "standard" goal metric... since I am stitching everything in my Products List, I can create breakdowns using those eVars... 

It's still some work setting things up, but maybe something similar could work for you?