How to organize and A/B test several newsletters a week | Community
Skip to main content
A
Anonymous
June 16, 2016
Question

How to organize and A/B test several newsletters a week

  • June 16, 2016
  • 2 replies
  • 3318 views

Hi! My company sends out around 3 newsletters a week. What I'm wondering is how to best format that so I can still properly A/B test each one? Would that necessitate creating a new program each time or should I just create a folder structure, clone the A-variant email for the B version and A/B test in two separate campaigns?

Thanks for any help!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

2 replies

Iryna_Zhuravel4
Iryna_Zhuravel4
Level 6
June 16, 2016

Hey Steve,

there are 3 methods to A/B test emails in Marketo:

1. A/B test in email program  - Email Test - A/B Test - Marketo Docs - Product Docs

2. Champion/Challenger in the email itself - Email Tests - Champion/Challenger - Marketo Docs - Product Docs

3. Random sample in the smart campaign - Some Tests on Random Sample Behavior

Here is my take on their pros, cons and preferable use cases:

1. A/B test in email program - Pros: fast and easy to set up, works best for one-off email blasts. Cons: a/b test data is only available in the email program, it won't show in the email performance reports

2. Champion/Challenger - Pros: easy to set up, works best for recurring/triggered emails. Cons: a/b test data is only available in the test, won't show in the email performance reports

3. Random sample in the smart campaign - Pro: data is available in analytics, arguably allows more flexibility than the other two mwthods. Cons: you will need to create 2 or more separate emails to use it

The simplest way to test your newsletters would probably be through an email program, but it all depends on your set up and how you prefer to analyze the data

A
Anonymous
June 16, 2016

This is great, thank you so much! So I guess it doesn't matter if Im declaring a champion anyway.

Why then would you think the email program would be best for me considering these are individual emails I have to build for each blast, but involves utilizing a single smart campaign?

Iryna_Zhuravel4
Iryna_Zhuravel4
Level 6
June 16, 2016

It all depends on how you are using these newsletters and what you are trying to achieve with A/B testing.

If you have an email that you are planning to send just once, and you want to get the most out of it, use the email program. Send the test to a small percentage of the target audience, see which version performs better and send it to the rest of the target group. You don't have to create two separate emails, and you can make the program to declare a winner automatically.

If your newsletters are part of a nurture stream in the engagement engine and are being sent on the regular basis, the easiest way to A/B test them would be through Champion/Challenger. Let it run for some time and then declare the champion.

Method 1 and 2 are intended to improve the performance of an individual email, so you would have to declare a winner once you have enough data.

If you want to test a hypothesis, let's say you want to see what works better in a subject line - a statement or a question or something like this, use the random sample. You will have to create 2 emails, but you can then pull reports on all the emails tested through the random sample and analyze the data.

Let me know if it makes sense.

Dory_Viscoglio
Dory_Viscoglio
Level 10
June 17, 2016

Hey Steve, regardless of how you end up setting up the actual email assets, I wanted to include one small piece of information that relates to newsletters/consistent sending of something to the same group of people.

If you don't use email programs, be sure that for each send to people you use a new smart campaign. The reason for this, is that Marketo will only track the send of something to someone the first time in a smart campaign. We had an issue where we tried to have 7 programs, 1 for each day of the week, and we set the reporting timeframe to just the past week. The first week everything looked fine, but the second week the only results that we saw were for people who happened to be new to the smart campaign that week, because Marketo wouldn't report on people who it had already reported on. It sounds a little strange, but the end result was that we had to create a new smart campaign for each day's send. Let me know if you have any questions on this.


Also, my two cents: I always use smart campaigns and random sample, with two unique assets. It allows me to look back and see the test that we ran, as well as the results -- forever. Email programs have given me some issues in the past (that I'm sure have been resolved, but once I've been burned I tend to stay away), and champion/challenger tests lose their data once you declare a winner.

    Iryna_Zhuravel4
    Iryna_Zhuravel4
    Level 6
    June 17, 2016

    Good points, Dory!

    I also prefer to do my a/b tests through the random sample, email programs didn't give me any major issues, but I see them more as a quick and dirty way to send one off emails

    Terms & ConditionsAccessibility statement

    Loading footer...