What we’ve learnt from randomly testing #EmailWeekly

What we’ve learnt from randomly testing #EmailWeekly

If you receive the #emailweekly regularly, you have been subject to innovative/ridiculous designs, delayed sends, obscure subject lines and the odd error here and there. Some of these were on purpose, others weren’t. I guess you could say we’ve kind of learnt something from each one.

We like to think of the weekly as a playground for email code so we thought we would bare all for the eyes of our subscribers. Here is what we’ve found from AA/AB testing/sending the email weekly so far:

A/A tests (on purpose)

EmailWeekly #29

Screenshot 2015-09-30 10.26.36

Subject line: GOOD EMAIL ¿ pheasant! downpour! schist! aaaaa
Send time: 
Declare winner: 
1 hour
Overall open rate:
Grammar mistakes: 1

Screenshot 2015-09-30 10.22.33

Variance: 11.7%

EmailWeekly #30

Screenshot 2015-09-30 10.34.01

Subject line: EmailWeekly #30: Keeping it alternative
Send time: 
Declare winner: 
1 hour
Overall open rate:
Grammar mistakes: 1

Screenshot 2015-09-30 10.33.21

Variance: 9.7%

EmailWeekly #43

Subject line: EmailWeekly #43: Lots to report
Send time: 12:22PM
Declare winner: 1 hour
Overall open rate: 45%


Variance = 3.7%

EmailWeekly #44 


Subject line: Hey there [firstname,fallback=you], just testing subject line length, don’t mind me. Enjoy these articles and I’ll see you next week
Send time: 1:20PM
Declare winner: 1 hour
Overall open rate: 47%


Variance = 3.3%

EmailWeekly #45


Subject line: Time for a #cheekyemail
Send time: 1:10PM
Declare winner: 1 hour
Overall open rate: 48%


Variance: 3.8%

Lesson learnt? Subject line B will always win, most of the time.

Other tests:

EmailWeekly #7

Screenshot 2015-09-07 12.33.12

Admitting you are testing your subscribers… but aren’t actually. This is one of our best open rates so far.

Subject line: [Subject line split test A: this group are much nicer than group B]
Send time: 6PM
Overall open rate: 54%

EmailWeekly #58

Screenshot 2015-09-07 12.12.57

Burrito (version B) vs Burger (version A) in the subject line…

Subject line: do the subject line later, it’s burrito Friday
Send time: 4pm
Declare winner: 1 hour
Overall open rate: 44%

Screenshot 2015-09-03 10.38.42

Variance: 2%

EmailWeekly #57

Screenshot 2015-09-07 12.16.00

Subject line: Subjectline
Send time: 3pm
Observation: We saw a 6% increase in open rate from the week before and week after. We’ll put that down to curiosity
Overall open rate: 48%

EmailWeekly Trivia
Our highest CTR: EmailWeekly #37. The mystery CTA of course.
CTR: 60%

Screenshot 2015-09-07 13.04.50

Our lowest ever open rate:

EmailWeekly #26

Screenshot 2015-09-07 12.49.42

Subject line: Decorations? Check. Presents? Check. #EmailWeekly newsletter? CHECK.
Send time: 5pm
Open Rate: 39%
Probable insight: This was sent at 5pm on the Friday before Christmas, most people have probably logged off by this point to spend time with their families if they aren’t already on holiday. We can only assume.

To sum up…

We’ve learnt that our own random variance stays at about 3% (excl. the outlier), email marketers prefer burritos over burgers (just) and long subject lines are pretty awesome, don’t knock them. We’ll continue with our random tests so you don’t have to.

Until next time.

  • http://www.rajeshtaylor.com/ Rajesh Taylor

    With email weekly #7 were you also testing the overlaying colours too for click through aswell? 44, 45 and 48 have one of the top 3 images close to the colour in the banner. Be interesting (seriously) to see a test on colour click through’s.