We Split Tested a Nonsense Subject Line, You’ll Never Guess What Happened Next!
Well, actually not much. But that’s because we already know the results of this test are useless.
Last week Clickhole posted this excellent send up of subject line “best practice”, including such gems as ‘Start your subject line with “GOOD EMAIL”‘ and ‘For urgent emails, write “I HAVE YOUR DAUGHTER”‘. We’ve been questioning the value of perceived subject line best practice for a while – there’s obviously a lot of value in writing a subject line that grabs your audience, but we’re not convinced that following an arbitrary set of rules is the right way to get there.
So we decided to run a split test on our weekly news roundup, #emailweekly. Our first subject line being “Perfect Subject Line Tips + Top Email Thought Leaders”, and our test line being “GOOD EMAIL ¿ pheasant! downpour! schist! aaaaa”. And you know what? maybe Clickhole are onto something, because the test out-performed our regular subject line 2 to 1.
What can we learn here? well the lazy answer would be to put “GOOD EMAIL” at the start, use a ¿, and use pretty words like “pheasant” and “downpour”. Oh, and don’t use too many A’s. But that’s obviously nonsense.
It’s likely that if we did the same next week we wouldn’t get the same result, so why did the Clickhole version perform better? It could be because it stood out against other emails for being nonsense. Maybe it looked like a mistake. Maybe it looked like we’d screwed up. Maybe our alternative was just too boring. Maybe some people had already read the Clickhole article. Maybe people saw that I’d tweeted in advance and decided to skew the test on purpose.
There’s so many “what ifs” that it’s hard to draw any kind of reliable conclusion from the data – indeed the reasons are likely different for different members of our audience too. Even if we could get a useful conclusion, it would still only apply to this given email, this given day, this given audience.
If we were to behave like robots, we’d consider sending a similar subject line again, but as humans we know that sending subject lines that at least make sense is probably a good idea. So maybe in future we should just follow our intuition and write lines we think our audience of fellow humans will like?