Udemy + UserTesting

As the world’s largest online learning and teaching marketplace, Udemy is committed to constant improvement and lifelong learning. And this is true internally as well. Notes Janel Faucher, Senior UX Researcher, “Udemy has a company culture that promotes curiosity. We operate from a very learning-driven, empathetic mindset. Udemy employees want to hear what our users have to say.”

This has afforded the research team much influence over the evolution of Udemy products. “The research team is part of our product team, which puts us in the conversation about our product roadmap.”

Through data analysis, the research team knew that there were differences in student purchase and consumption patterns depending on device. The research team wanted to better understand the students who used Udemy mobile apps, especially any behavioral differences between people who only used mobile, and those who learned across both desktop and mobile devices.

Michelle Fiesta, UX Researcher, explains, “We wanted to better understand how students use both desktop and native app experiences. We knew that some students switched between devices while learning, and we wanted to discover more about that group, particularly as compared with those who only used mobile apps.”

She adds, "We wanted to observe our users over a longer period of time, noting behaviors as they happened to determine how/if these user groups were different, which wouldn't have been possible with a one-off interview. "

To gather more natural, in-the-moment feedback, the team decided to employ a diary study to better understand how their learners use Udemy over time. They looked into a number of companies that offered specialized diary study products, however found them all to be cost-prohibitive. In addition to cost, time also proved to be a limiting factor. Notes Faucher, “We were unable to find a separate solution that was cost effective and could support our timeframe.”

A longtime UserTesting customer, the Udemy team decided to create a diary study using remote unmoderated testing. Faucher adds, “Being a digital product, as well as a global company, we rely on remote research. We’re able to capture the voices of customers anywhere, all over the world. All things considered, remote research is the most cost-effective way for us to get as much quality insight as possible.”

As co-researchers, Faucher and Fiesta executed the study as several ‘missions,’ or remote unmoderated tasks over a week, each with its own focus. Fiesta explains, “Basically, each time a student used Udemy, we asked them to use the camera on their mobile phone to show us where they were and what they were doing, including their environment and the context around that specific moment.”

The team drafted email reminders for each day’s assignment to keep contributors on track. By creating all content pieces ahead of time, they were able to quickly send out emails each day by simply copying and pasting.

The team also created short instructional videos, shot using a mobile phone, which they shared with contributors prior to each ‘mission.’ Explains Faucher, “The videos were really meant to reinforce the instructions and set expectations. They also helped to break the sometimes awkward barrier that can occur between researchers and contributors, especially with remote research. It allowed them to see our faces and know there were real human beings on the receiving end of the study.”

She adds, “Even though they were low cost and shot all in one or two takes, we received a lot of compliments on the videos. People were really excited to see them.” The videos were such a success that the team now intends to repeat this step in future studies. The team stayed organized using shared spreadsheets and kept studies organized in UserTesting. Faucher concludes, “UserTesting helped us establish a framework for running this study more quickly and effectively. Completed studies were centralized in a single location, and team members could take notes and create highlight reels when reviewing responses.”

Giving customers the freedom to complete the study whenever a learning moment struck yielded interesting insights into how and why students opted to use mobile devices. Explains Fiesta, “When people say ‘mobile,’ most people think about that in terms of movement—as in you’re moving, you’re on the go, you’re holding it in your hand.

“However, we found that when people were learning while on their mobile device, they were actually stationary and in one place— like sitting on their couch at home or eating lunch in their work cafeteria. For some students, it just happened to be more convenient and more desirable to be on a mobile device rather than use their desktop or laptop computer."

She adds, “The shift in mindset and the understanding that there is a nuance in how students use mobile is an important realization for Udemy. As a company, we had talked about mobile in the context of ‘mobile versus desktop.’ But really it’s not binary—it’s more of a spectrum. This enables us to think about mobile users through a new lens.”

Business stakeholders are now using the insights for next year’s product and feature planning. Adds Faucher, “We brought the business in at the onset, to make sure that would be something that they would find useful at the completion.”