How long after an email was delivered do you report metrics? | Community
Skip to main content
Matt_Rodriguez
Matt_Rodriguez
Level 2
November 27, 2018
Question

How long after an email was delivered do you report metrics?

  • November 27, 2018
  • 1 reply
  • 3885 views

Hi Marketo community,

I'm wondering what the best practices are around when to start reporting on email metrics. My hunch is that 48-72 hours is enough time for the majority of behavior to be captured, but does anyone have hard stats around the best timing? Additionally, how long should I be running subject line and CTR A/B tests? I've been running them for 7-10 days at a time but would love any additional insights here.

Thanks,

Matt

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

    1 reply

    Grace_Brebner3
    Grace_Brebner3
    Level 10
    November 27, 2018

    I am struggling to find resources to share that back this up (I'm sure I've seen it validated in a report on litmus somewhere...) but my experience, and I suspect this is consistent with most others, is that the vast majority of engagement happens within 24 hours of the email being sent. But like everything there's probably some variation here.

    As for running tests; I run them for as long as is required to reach a statistically significant result. For me this is a pretty major flaw in Marketo's A/B and Champion Challenger testing functionality - there's no statistical significance calculator and it's easy for marketers to make poorly informed decisions here without some knowledge on stats. There's great statistical significance calculators online, and basically I stick with a >95% model on everything. Depending on the size of your audience + test groups, and on your success metric, the time it takes to reach statistical significance may vary. If your success is based on email engagement, if you haven't reached statistical significance within 24 hours, you're probably unlikely to reach it without another send.

    Matt_Rodriguez
    Matt_RodriguezAuthor
    Level 2
    November 27, 2018

    Thanks Grace! 24 hours makes sense to me and so 48 hours should be sufficient as well. I've run a dozen tests in the last few weeks but only 3-4 of them have come up as significant (using 95%). Is it more likely that my tests just aren't different enough to garner significant results? I can't imagine running a subject line test for a full month just to be told that the champion was already better-written.

    Grace_Brebner3
    Grace_Brebner3
    Level 10
    November 27, 2018

    Yeah if 48 works for you and you're not in a rush

    There could be a number of reasons why you haven't got many statistically significant results; it could be that your sample sizes are too small (thinking not just about your sending audience but your success metric), it could be that the differences are too minor/insignificant to have impact, or it could just be an interesting example of things that are both performing equally as well (or poorly!) as each other.

    When my users are running A/B or Champion/Challenger tests, I always encourage them to test things that are meaningful - that have potential for longer term impact on our ways of doing things, and be strategic about it. That could mean running multiple tests with effectively the same objective - say, do long subjects perform better than short ones - to do our best to control against other factors that could influence the test. But finding the things that people ​don't ​care about, that ​don't​ impact performance - that can still be a good learning!

    Terms & ConditionsAccessibility statement

    Loading footer...