A/B testing in Stream, Is BOT activity driving statistically unremarkable results? | Community
Skip to main content
hlawrence
Level 2
February 20, 2024
Question

A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

  • February 20, 2024
  • 1 reply
  • 2228 views

This question references my previous question on BOT activity

 

We are engaging in some A/B testing of our marketing emails that are in an engagement stream, as one should.  I choose "new test" 50/50 with champion and challenger.  Up to now we have only done subject line tests.  I have tried 'Click to open %' and 'Custom Conversion Ratio' (clicking on one specific link in that email), but all results are within a couple clicks of 50/50. 

 

This week we tried whole emails, and STILL the results are 50/50

 

I am starting to get the feeling that BOT activity is driving these A/B test results.  In the previous question linked above, we had found out that a large amount of our clicks were coming from the first link in the TEXT ONLY version, suspecting BOT activity.  A recent email:

Do you have any ideas on how to filter out BOT clicks for A/B testing?

Thanks.

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

1 reply

Vinay_Kumar
Community Advisor
Community Advisor
February 21, 2024

Have you enabled the Bot Activity Identification in your instance?

If it is enabled then you use a smart list to check the bot clicks.

 

 

hlawrence
hlawrenceAuthor
Level 2
February 21, 2024

This is how our BOT activity Identification is set.

 

Zoe_Forman
Community Advisor and Adobe Champion
Community Advisor and Adobe Champion
February 21, 2024

As @vinay_kumar says this looks to be set up correctly,

 

@hlawrence have you created the Smart lIsts to identify BOT activity?