Skip to main content
hlawrence
Level 2
February 20, 2024
Question

A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

  • February 20, 2024
  • 6 replies
  • 2254 views

This question references my previous question on BOT activity

 

We are engaging in some A/B testing of our marketing emails that are in an engagement stream, as one should.  I choose "new test" 50/50 with champion and challenger.  Up to now we have only done subject line tests.  I have tried 'Click to open %' and 'Custom Conversion Ratio' (clicking on one specific link in that email), but all results are within a couple clicks of 50/50. 

 

This week we tried whole emails, and STILL the results are 50/50

 

I am starting to get the feeling that BOT activity is driving these A/B test results.  In the previous question linked above, we had found out that a large amount of our clicks were coming from the first link in the TEXT ONLY version, suspecting BOT activity.  A recent email:

Do you have any ideas on how to filter out BOT clicks for A/B testing?

Thanks.

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

6 replies

Vinay_Kumar
Community Advisor and User Group Leader
Community Advisor and User Group Leader
February 21, 2024

Have you enabled the Bot Activity Identification in your instance?

If it is enabled then you use a smart list to check the bot clicks.

 

 

hlawrence
hlawrenceAuthor
Level 2
February 21, 2024

This is how our BOT activity Identification is set.

 

Zoe_Forman
Adobe Champion and Community Advisor
Adobe Champion and Community Advisor
February 21, 2024

As @vinay_kumar says this looks to be set up correctly,

 

@hlawrence have you created the Smart lIsts to identify BOT activity?