Something that I like to talk about when the topic of email testing comes up is the idea that open rates — while the email statistic people are often the most familiar with — are pretty much never the thing you want to be optimizing for.
Why? Simple: You don’t send an email to get people to open it. You send an email to get people to take an action (or, at least, you should!). And the fact is, worrying about open rate too much is not just misguided — it can also be counterproductive.
Here’s a great case in point from recent experience
A couple of weeks ago, for our penultimate appeal as part of our anniversary dinosaur campaign, we did subject line tests on our email.
Splitting our audience of recent donors (past 12 months) into three random test groups, we tested the following subject lines to 5,343 people each:
1) Victory Lap
2) Last chance to get your dino
3) Celebrating with all of you
Here were the open rates for each segment:
1) 49.1%
2) 40.9%
3) 45.7%
But, click rates were:
1) 3.5%
2) 3.8%
3) 3.1%
And here’s the number of donations from each segment — the action we really care about:
1) 43
2) 66
3) 53
So not only did the one doing much worse on opens win in donations, subject three, which was worse than subject one on both opens and clicks outperformed one on the metric that mattered.
It is worth flagging that the win on donations didn’t quite reach the 90% confidence level I’d like, but the broader point certainly still stands.
