Prediction Score for Auto Target AI Algorithm

[emmak109]

Hi all, I have a question from the reporting side. When we share AT findings with the stakeholders, they want to know the accuracy of these models for every optimal experience identified for a segment. In AB activity, we have Confidence (pvalue T-Test) to understand the impact. Which accuracy score do we use in the case of Auto Target? 

[Devyendar]

@emmak109 

AT (Auto-Target) delivers personalization using ensemble machine learning models (logistic regression + decision trees + bandit learning) to predict the best experience per user based on behavioral and contextual signals (not a fixed A/B/n split), the classic "confidence" (p-value between variants of experiences doesn't apply. Adobe suggest to evaluate the outcome based on uplift vs. control with confidence rather then a stand-alone classifier accuracy.

Refer to:

Auto Target reporting: https://experienceleague.adobe.com/en/docs/target/using/activities/auto-target/reporting-and-auto-ta... 
Auto-Target Overview: https://experienceleague.adobe.com/en/docs/target/using/activities/auto-target/auto-target-to-optimi... 

[emmak109]

Hi Adobe, 

Since each experience optimizes by automated segments, and when that experience shows a higher conversion for a specific segment, we would like to know how confident we can be on that model performance that worked for a specific segment.

Perhaps looking at precision and recall score would help, however, since it's not available here, what else could we lean on to ensure accuracy of the model performance across individual experience? 

[Devyendar]

@emmak109 I hope my response to your question was helpful, please mark it as "Correct Answer" so it can help other community members as well.