Split-Testing Responsive Email Design: You’re Doing it Wrong
if you're split testing responsive vs. non, and looking at clicks or opens, you're doing it wrong #emaildesign
— Elliot Ross (@iamelliot) July 10, 2014
I watched a webinar yesterday where the presenter was strongly advocating doing a split test between a Responsive email vs. a Non-Responsive one (ideally the same email just sans-media queries) and I was reminded of a rant I half-wrote on my flight back from the Litmus Conference last year. There’s still too many swear words in that, but nevertheless I still think this approach demonstrates a lack in understanding in the benefits of going responsive.
Responsive email design is an experience thing. It’s about making the experience for a user better when they’re using a mobile device, and that’s a very difficult thing to quantify using numbers. Any difference in opens, or clicks, can be down to so many things, that they’re at best only a vague indicator. Even if you look at conversions, there’s so many factors at play that it isn’t a reliable statistic.
The true measure of things like responsive design, is to gauge the user’s comprehension of the message and how happy they are with the brand as a result — but whilst that’s perhaps the ultimate aim in marketing, that’s incredibly hard to put a number on.
Incidentally, towards the end of my flight on that trip home, the flight attendants handed out free choc-ices to everyone. That wasn’t part of the in-flight meal — that was long gone — so what was the return on investment on that? could you split test that and see who re-books another flight? No you can’t. There’s no reliable correlation there whatsoever. But I’ve still flown with Virgin Atlantic three times since then.
- Why should a business go responsive?
- Is responsive email design really worth it?
- WhichTestWon: Responsive vs. Non Responsive
Image: Ivan Colic / Noun Project