The Fascinating Things You’ll Learn From Mobile Testing Your Emails

In this guest post, usability expert Jenn Downs (who pioneered user research and usability testing for MailChimp as well as created a multi-platform testing process for their in-house products) explains how you can effectively test your emails on mobile devices.

As researchers, you know you have to test your websites for usability and your workflows for clarity, but have you ever thought about testing your marketing emails?

Sure, you may currently test the design and code of a marketing email in multiple email clients before sending to your subscribers, but have you ever actually watched someone read your marketing emails?

It’s really fascinating to dig into!

Know Your User Demographic

The first thing you want to find out about your marketing emails is – do you have a high mobile readership?

It’s reported that 43% of opens happen on mobile devices, so don’t be surprised if you do! We’re seeing such great open rates on mobile that we decided to do some testing specifically on mobile emails.

During our initial studies on mobile email, we observed 40+ people reading emails on their mobile devices using Skype screen sharing or the Laptop Hugging Method.

How We Set Up Our Test

To prepare for this test, we asked participants to hold off on checking and opening any emails until an hour or two before we planned to meet. You’ll be lucky to get folks to resist opening emails for even that long – 76% of people in our study said they view their inbox as soon as they get alerts! During this excruciating hour or two without email we would send out our curated mix of test emails.

While we wanted to go through a handful of specific emails with our test participants, asking them to save up all their morning’s emails would allow us to see how our test emails looked in their inbox, surrounded by other important emails. This let us see how our “from names”, subject lines, and preheader paragraphs looked next to other emails. And it allowed us to observe our participants looking over emails that we didn’t handpick, to get a broader sense of their mobile reading habits.

Here’s a sample of the questions we asked during our tests to get a deeper look into email behaviors:

• Do you use your device for personal or for work emails? [Internal Notes – “Both” is a valid answer.]
• Do you check your email before you get out of bed in the morning?
• Where’s the weirdest place you read your email? [Internal Notes – If they hesitate, offer an example that suggests weird behavior is common. For instance, “I sometimes snuggle with my dog with one arm under him and rest my phone on him and navigate with my free hand.” Do not use the bathroom example, it’s too common and not weird anymore.]
• Do you use the built in mail feature to read your emails (ask them to show you)? [This will transition the interview to reading the emails. If no, ask what app and start reading emails.]
• Do you shop from mobile emails? [Internal Notes – Ask on an email where they are seeing links to purchase items.]

During the interview, you can see we set up a seamless transition from background questions to watching participants read actual emails they received by asking “Do you use the built in mail feature?” and prompting them to start looking at their emails. Then we’d listen for feedback and watch their gestures and interactions with their emails on mobile devices.

What We Discovered from Our Tests (and What You Can Expect to Discover from Yours!)

When you watch people read your marketing emails, you can not only find out how your subscribers read and interact with your own emails, but while you’re at it also find out some of your users’ general habits.

How many of your readers say they check their email in bed on their mobile device? 72% of people we spoke to read their emails in bed, and 77% of people said they check email “everywhere” or “obsessively”. How many save it for later? (Probably not as many as you think.) Are people missing your call to action because it’s a tiny link they can’t tap with their thumbs? (Make it a button instead!)

One of the funniest, and seemingly cliché, things we discovered is that readers over the age of 42 do have a harder reading smaller text. Luckily, the Apple recommended font size of 17-22pts in mobile emails satisfies most mobile readers.

Our full report on mobile email findings, as well as some mobile email design and style recommendations, is available to read online or download (maybe to your mobile device?).

___

If you have any questions or thoughts about mobile testing, please ask me or share! You can also follow me @beparticular or @mailchimpux on Twitter or attend a talk / workshop I will be offering in your area. I’ll be presenting a workshop on Usable Findings and talking about how Empathy Cannot Be Automated at UX Week in San Francisco Aug 20 – 23 and at ConvergeFL in Jacksonville, FL Sept 11-13. http://convergefl.com/

Hope to see you there!