January 30, 2015

We Split Tested a Nonsense Subject Line, You’ll Never Guess What Happened Next!

We Split Tested a Nonsense Subject Line, You’ll Never Guess What Happened Next!

Well, actually not much. But that’s because we already know the results of this test are useless.

Last week Clickhole posted this excellent send up of subject line “best practice”, including such gems as ‘Start your subject line with “GOOD EMAIL”‘ and ‘For urgent emails, write “I HAVE YOUR DAUGHTER”‘. We’ve been questioning the value of perceived subject line best practice for a while – there’s obviously a lot of value in writing a subject line that grabs your audience, but we’re not convinced that following an arbitrary set of rules is the right way to get there.

So we decided to run a split test on our weekly news roundup, #emailweekly. Our first subject line being “Perfect Subject Line Tips + Top Email Thought Leaders”, and our test line being “GOOD EMAIL ¿ pheasant! downpour! schist! aaaaa”. And you know what? maybe Clickhole are onto something, because the test out-performed our regular subject line 2 to 1.

Split testing that Clickhole advice

A photo posted by ActionRocket (@actionrocket) on

What can we learn here? well the lazy answer would be to put “GOOD EMAIL” at the start, use a ¿, and use pretty words like “pheasant” and “downpour”. Oh, and don’t use too many A’s. But that’s obviously nonsense.

It’s likely that if we did the same next week we wouldn’t get the same result, so why did the Clickhole version perform better? It could be because it stood out against other emails for being nonsense. Maybe it looked like a mistake. Maybe it looked like we’d screwed up. Maybe our alternative was just too boring. Maybe some people had already read the Clickhole article. Maybe people saw that I’d tweeted in advance and decided to skew the test on purpose.

There’s so many “what ifs” that it’s hard to draw any kind of reliable conclusion from the data – indeed the reasons are likely different for different members of our audience too. Even if we could get a useful conclusion, it would still only apply to this given email, this given day, this given audience.

If we were to behave like robots, we’d consider sending a similar subject line again, but as humans we know that sending subject lines that at least make sense is probably a good idea. So maybe in future we should just follow our intuition and write lines we think our audience of fellow humans will like?

  • Nick L

    I can’t see the image – was there a noticeable difference in the click-through rate between the two sends?
    I would open any weird subject line – especially if it promises pheasant downpours.

    • http://www.actionrocket.co Elliot Ross

      not much – 4% but that’s probably within acceptable variance

  • Jaina

    There are too many what ifs to really be able to predict whether a subject line or a piece of copy will be enough to illicit a reaction out of someone. All any email marketer can hope to do is cover as many bases as possible.

    Oh and PS – disappointed I didn’t get a pheasant in my subject line 🙁

    • http://www.actionrocket.co Elliot Ross

      hehe I’ll see if we can fix it next time 🙂

    • Sam Sexton

      Yeah, there are so many variables here, although, thinking about this yesterday I realised that it kind of depends on the strength of your brand. Brands I trust really just need to tell me whats inside, whereas weak brands need to provoke me in to opening.

      • Jaina

        Yep, the strength of your brand and how good a relationship you have with your customers are massive uplifts in terms of opens. I too will automatically open an email from brands I know and trust as well as brands I enjoy receiving emails from. Amazon’s up there on this – rarely ever look at the subject line for Amazon emails, just instantly know that when I open them, there might just be something I might want because they’re so damn good with their personalisation.

  • Dela Quist

    Hi Elliot

    Thought you might be interested
    to know that Touchstone http://www.subjectlinegold .com would have predicted the outcome of the test without you having to do it!

    Touchstone predicted that

    “Perfect Subject Line Tips Top Email Thought Leaders”: would give an open rate +21% above average

    “GOOD EMAIL ¿ pheasant! downpour! schist! Aaaaa”: would give an open rate +41% above average

    In your tests the nonsense SL outperformed the sensible one 2 -1 in and that’s almost exactly what Touchstone predicts (41 : 21)!

    Touchstone also predicts that the nonsense SL would have a much worse click rate – is that what happened? If so you probably would not have bothered to test the nonsense SL had you been using Touchstone. Give it a try!

    Dela

    • http://www.actionrocket.co Elliot Ross

      will take a look – looking at the stats there’s a 4% difference in clicks, where the sensible one delivered more clicks.

  • Abigail Wentzel

    I’m just so excited to get my EmailWeekly from Action Rocket that I bypass the subject line entirely! That said, I went back to see if I got the test email. I did not, and am now sorely disappointed… 🙁

    • http://www.actionrocket.co Elliot Ross

      haha next time!