False open rate results? | Community
Skip to main content
A
Anonymous
January 14, 2015
Solved

False open rate results?

  • January 14, 2015
  • 4 replies
  • 2373 views
I ran just ran a split test with my winner critera/CTA as a call in, with the body content of the email was different. There was no difference in the number of call-ins but I did notice that the Challenger provided a 3% increase in open rate. This is of course a good thing, but I'm stumped as to how or why I got these results, as the From name, From Email, Subject line, and Preheader were all identical.

One difference is that the Challenger had an image in the body content removed. I understand that opens are tracked when someone downloads images. Is it possible that the lack of images in the body just encouraged more people to download images, creating a false lift in my open rate? Or any other ideas on how this could happen is appreciated!
This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by

Hey Manisha > 

Perhaps 3% more of people in the Challenger group just opened the email? I don't think there's any conclusion I'd draw from the set of data you put forth. I'd test this image idea another 10 times before I'd draw any conclusions.

Changes in the body of an email don't change open rates (**unless it effect deliverability**)

4 replies

A
Anonymous
Accepted solution
January 14, 2015

Hey Manisha > 

Perhaps 3% more of people in the Challenger group just opened the email? I don't think there's any conclusion I'd draw from the set of data you put forth. I'd test this image idea another 10 times before I'd draw any conclusions.

Changes in the body of an email don't change open rates (**unless it effect deliverability**)

A
Anonymous
January 14, 2015
I'm new to testing - how often do identical assets get varied results like that? Out of curiousity I plugged it into a statistical significance calculator, and it's 99% confident that the challenger won in terms of Opens. Deliverability doesn't seem to be an issue either, as delivery rates are similar as well.

You're right - I will definitely test more before applying any sort of changes. But the thought that I could get a false increase in opens due to number of images is interesting to me, and was wondering if this was a "thing" I should be thinking about.
Josh_Hill13
Josh_Hill13
Level 10
January 14, 2015
It may be possible that your image change modified what was seen in Preview of the email reader, and encouraged more image downloads. I wouldnt use open rates for anything as they are unreliable due to the image blocking software.
SanfordWhiteman
SanfordWhiteman
Level 10
January 15, 2015
As @Adam W said, if content changes end up affecting deliverability, they will in turn affect openability.

A low ratio of text to image area is considered spammy (error-prone or not, this relationship is coded into major anti-spam software, based on the practice of hiding text within large images).  So if by reducing the # of images, you change the text:image ratio from, say, 2:1 to 6:1, you will likely see more deliveries.
Terms & ConditionsAccessibility statement

Loading footer...