The UX research methodology guidebook

Before we jump into individual research methods, let’s cover the big-picture questions about your research needs.

Quantitative and qualitative research

As you conduct your research, you’ll probably use quantitative and qualitative methods to answer your questions. Quantitative research methods rely on using large sample sizes to establish trends. Qualitative research, on the other hand, is appropriate for getting a more in-depth, contextual understanding of why those trends occur.

The two types of research answer different questions, so you shouldn’t substitute one for the other. To get a complete picture of your user experience, you must understand what’s happening and why. 

If you only have quantitative data, you may need to include key insights that could make all the difference in your understanding of the user experience. And if you only do qualitative research, you won’t be able to tell whether your findings are representative of a larger population. That’s why the best research strategies incorporate both approaches.

Attitudinal and behavioral research

Attitudinal research involves the assessment of users’ preconceived attitudes or feelings toward an experience. For example, this could involve asking a user how they feel about a product or brand based on its reputation. 

By contrast, behavioral research is focused on what the user does. Keep in mind that what users say and what users do are often different. For example, someone may say they have a positive impression of a product but never use it.

Stage of the product life cycle

The type of research you do will depend on the stage of development of the product you’re interested in. Each stage of development has different research objectives and questions that need to be answered.

Before development

Goal: Exploration
Questions: What do people need? What are they already using? What’s working and what’s not? Where are we starting?
Methods: Interviews, focus groups, longitudinal studies, 1:1 user interviews, competitive analysis, benchmark studies

During development

Goal: Evaluation and validation 
Questions: Are we on the right track? Does the product look and feel right? Do users get how to use it? Can they find what they’re looking for?
Methods: Card sorting, tree testing, moderated and unmoderated usability testing, preference testing, prototype testing

Live design

Goal: Measurement
Questions: How are we performing? What can we optimize? Does our actual user experience meet users’ expectations?
Methods: Moderated and unmoderated usability testing, A/B or multivariate testing, benchmark studies, longitudinal studies, diary studies, multichannel and omnichannel studies

Many UX research teams use office space in-house to conduct in-person studies, which can be great for longer usability tests, interviews, and focus groups. But not all research needs to be conducted in person to be helpful.
Many methods can be done remotely, meaning your pool of potential study participants isn’t limited to a location. You can often conduct more research sessions in a shorter time and with less recruiting hassle if you do remote research.

Remote user research, where participants share their feedback from their location, can give you more realistic insights than lab research. Since participants aren’t in an artificial and unfamiliar setting, they’re likelier to give honest reactions and unbiased opinions. You also gather contextual findings about how your product is used naturally.

Plus, some research just can’t be done in a lab. Suppose you’re trying to answer questions about how users interact with a product in a specific context, for example, using a fitness app at the gym or setting up a smart home device in their house. In that case, you’ll need to ensure users can participate in your study from that location.

Remote unmoderated usability studies

In a remote unmoderated usability study, users complete a series of pre-written tasks in their own location with their own device. Participants record their screens and speak their thoughts aloud as they complete the tasks, and the researcher reviews the recorded sessions at a later time. The researcher doesn’t communicate with the participant in real time. The researcher may include follow-up questions as part of the study, or they may follow up with the participant after viewing the session.

When to use it

Remote unmoderated usability studies can be used at any stage of the development process. They are great when you’d like to:

• Gather quick feedback: You can get results within hours since no scheduling is required.
• Test with larger sample sizes: Since a moderator doesn’t need to be present during each session, you can include as many participants as you like.
• Avoid bias: With no moderator present, participants may feel more inclined to be open and honest.

Limitations

• No guidance: Especially if you’re testing prototypes that aren’t fully functional yet, there’s a chance users might get confused or sidetracked.
• Fixed tasks and questions: Participants stick to the script the researcher wrote, so there’s no opportunity to dig deeper or change direction like in a moderated study.

Remote moderated usability studies

In a remote moderated usability study, a moderator meets virtually with participants to guide them through a pre-written test. The participant typically thinks out loud as they complete the tasks. The moderator observes the user as they work through the tasks, and the moderator may ask questions for clarification or to gather more data.

When to use it

Remote moderated usability studies can be used at any stage of development. They are great when you’d like to:

• Allow for unique, individualized follow-up questions in real-time.
• Ensure participants don’t get stuck while doing something technically demanding, like dealing with a complex or low-fidelity prototype.
• Ensure quiet participants are more vocal since the moderator can prompt them to share their thoughts.

Limitations

The drawbacks of remote moderated tests include the following:

• Less scalable: They require moderator time, which can limit the number of sessions you can reasonably conduct.
• Risk of bias: The presence and demeanor of even an experienced moderator can sway participants’ actions and responses.

Prototype testing

Prototype testing involves sharing prototypes or wireframes with participants to view on their desktop or mobile device. Testing your prototype with users lets you iterate based on feedback before any code is written, reducing your risk of an unsuccessful launch. It can be done in person or remotely, with or without a moderator.

When to use it

You can start testing as soon as you have an idea or wireframe and then continue to test as you revise your prototype. Getting user feedback early and often will derisk your product launch and help you avoid doing costly rework.

Pro tip

Let your participants know that they’ll be interacting with a concept rather than a working product so they don’t become confused or frustrated if links don’t work and interactions aren’t fully fleshed out. Remind them to express what they expect will happen next.

Competitor studies

A competitor study provides insight into what your competitors are doing well and where your competitive opportunities are. In a competitor study, participants interact with one or more of your competitors’ products. You can also have participants complete the same tasks on two products, yours and a competitor’s, and compare the experiences. They can be done in person or remotely, with or without a moderator. 

When to use it

Run a competitor study any time you want to observe the UX of a competitor’s product, such as when they launch a new feature or make a big announcement. These studies can also be particularly helpful when developing a new product or feature. Testing during this phase will help you identify competitive opportunities and advantages before your product is developed.

You can also test the competitor experience at regular intervals to keep a finger on the pulse of the competition.

Pro tip

While comparing your design to several competitors is tempting, remember that these tests can exhaust participants. To avoid fatigue and confusion, ask each participant to use and evaluate no more than two experiences.

Benchmark studies

Benchmark studies are a series of identical studies run at intervals to evaluate how your product’s UX changes over time. These studies use larger sample sizes than standard usability testing and always include a quantitative element. They also use the same core tasks and metrics each time to accurately compare one study to the next.

Because of the larger sample size needed, running benchmark studies using remote unmoderated testing is recommended.

When to use it

Benchmark studies are typically run regularly: monthly, quarterly, or yearly. They can also be helpful before and after rolling out a redesign or new feature.

Longitudinal studies

Longitudinal studies collect data from the same participants on multiple occasions over time. Studies can be run over a few weeks, months, or even years. A longitudinal study will provide valuable insight into how users behave and interact with their surroundings, an experience, or brands over time in their everyday lives. This study is an excellent tool for understanding how users interact with your product or service over time rather than just a moment in time. It can be done in person or remotely, with or without a moderator.

When to use it

Longitudinal studies can be done at any time. They can be especially helpful during the ideation phase before product development as a way to understand how users solve a problem without your product. You can also run them after development, which will give you insight into how, when, and why people naturally use your product or service.

Pro tip

Be sure to let your participants know that they’ll be conducting a series of tests, and be clear about the amount of time required to complete the study. Include a requirement in your instructions that notes participants will be expected to take a specific number of follow-up tests and that the test will last a specific amount of time. This can dramatically reduce the participant drop-off rate for this type of study.

Diary studies

A diary study is a type of longitudinal study where participants create “diary entries” as points of data. These diary entries can be audio, text, images, videos, or a combination of all and are typically combined with other studies to gauge participants’ expectations at the start and conclusion of a more extensive study. Diary studies can provide valuable insight into how users behave and interact with their surroundings or a product or service in their everyday lives. These studies are most effective when done remotely and unmoderated.

When to use it

Diary studies can be conducted at any time. They can be done before development, to gather requirements, or after a product has been launched to gather data on how, when, and why people naturally use it.

Card sorting

In a card sorting study, participants sort topics like pages, categories, or pieces of content into related groups. This can provide insight into how people group and label content and information in their minds. This is helpful when designing or assessing an information structure. 

There are two types of card sorts:

• Open card sorts: Users place items in groups and name the groups
• Closed card sort: Users place items into predefined categories

This type of study is most effective when done remotely and unmoderated.

When to use it

An open card sort is typically done in the early stages of the design process or to evaluate an existing design, and the findings can be used to inform the new design or redesign. Closed card sorts can be done before a redesign to see if the existing or proposed structure makes sense to users or when adding new content or functionality to see if it fits within the existing structure.

Tree testing

Tree testing is a way to evaluate the findability of topics within a website or app. Users are shown a text version of the site or app structure and are asked to indicate where they’d expect to find specific items or topics without the influence of the visual design of the navigation. Tree testing gives you insight into how easy or difficult it is for users to find essential items within a website or app. 

This type of study is most effective when conducted remotely and unmoderated.

When to use it

A tree test is usually conducted before the design phase to evaluate a current structure that will be redesigned or during the early design phase to evaluate a proposed structure.

Preference testing

In preference testing, participants interact with and review multiple designs to determine which one they prefer and why. Participants can compare aesthetics, interactions, and content. If you have more than one design option, this type of test can provide insight into which one stands a better chance of resonating with users. It can be done in person or remotely, with or without a moderator.

When to use it

Preference testing should be conducted in the early design and prototyping phases so that you have plenty of time to iterate without having to spend much time in rework later.

A/B and multivariate testing

A/B testing presents each unique user with one of two versions of a design—the A version or the B version—and conversion rates for each version are compared to see which one performs best. A variation of A/B testing is multivariate testing, in which you test three or more design versions.

A/B and multivariate testing are most valuable when only one element differs in each version. For example, using a different call to action for each design approach while keeping everything else the same allows you to draw firm conclusions about why one design performed better.

When to use it

A/B and multivariate testing can only be conducted on a live site or product. Unlike most other methods in this guide, A/B testing is strictly quantitative. Therefore, combining A/B testing with other methods is recommended to understand why users prefer one version.

Multichannel studies

In a multichannel study, participants complete the same tasks on more than one channel or device. There are two approaches to a multichannel study:

• Groups of participants complete the same tasks on one channel (e.g., desktop, tablet, or mobile). For example, Group A will complete the study on desktop, Group B on tablet, and Group C on mobile. This option is fast, but users only provide insight on a single channel.
• Each participant completes the same tasks on all channels. This option takes longer to recruit for and run. You’ll need more time to allow users to set up and switch devices, but it enables participants to compare and contrast the experiences across channels.

This is different from an omnichannel study, where participants complete an activity that spans more than one channel or device, like researching a family vacation on one device and booking it on another. 

These studies can be completed remotely or in person, with or without a moderator.

When to use it

Multichannel studies can be conducted at any time, but they’re particularly valuable when conducted before any redesign effort

Omnichannel studies

In an omnichannel study, participants complete an activity that spans more than one channel or device. For example, if a customer finds a recipe on their desktop, adds the ingredients to a shopping list on their smartphone, refers to the list on their phone while at the grocery store, and follows the recipe at home on their tablet, they’ve completed an omnichannel experience.

An omnichannel study is valuable when you want to understand how a single user experiences a brand, company, and organization across touch points.

These studies can be completed remotely or in person, with or without a moderator.

When to use it

Omnichannel studies can be conducted anytime, but they’re particularly valuable before redesign efforts. By doing so, researchers can identify what’s working well and what needs to be improved in the existing omnichannel experience.

Focus groups

A focus group involves a moderator asking people about their behaviors, preferences, attitudes, and experiences on a topic or item of interest. Focus groups can be conducted in person or remotely, and they require a moderator. Focus groups allow participants to interact and converse with others while discussing a topic and gathering in-depth details about a group’s actions, thoughts, and feelings. The brainstorming leads to people building on each other’s ideas, which allows a group to dig into an idea or issue. 

When to use it

Focus groups can be conducted at any time during the life cycle of a product. However, they’re typically done before development to gather requirements.

Surveys

A survey is a series of questions answered by the target audience. Responses are open-ended or multiple choice, typically about preferences, attitudes, and experiences on a topic or item of interest. The sample size must be large enough to capture statistically significant results to claim a user group.

When to use it

A survey can be conducted at any time during the life cycle of a product.

1:1 user interviews

1:1 interviews are conducted by asking participants questions and taking note of their feedback while facilitating further discussion. Interviews can provide in-depth details about a user’s actions, thoughts, and feelings without presenting a product. Like a moderated study, 1:1 interviews are often a good idea when probing for more information is necessary to receive in-depth reasoning from the test participant.
This can be done either remotely or in person.

When to use it

User interviews can be conducted at any time during the life cycle of a product. However, they’re typically done before development to gather requirements or after the product is launched to gather feedback on how people use it.

Once your research is complete, it’s time to take action! Your findings will help your company prioritize improvements and new features. But just as significantly, sharing your results with your team can help develop a user-centric culture in your organization.

Uncover findings and opportunities

Review your findings with the team responsible for the product or experience. Discuss what you learned and compare that with your hypothesis or previous findings.

• Did your research findings prove or disprove your hypothesis? Were there any significant surprises that will affect the direction of the project?
• What patterns emerged? Make a note of any usability issues, especially severe or recurring ones. Dig into any trends in your quantitative research, such as how participants perceive your brand or how likely they would be to recommend your company to a friend.
• How will those patterns affect your company? Which stakeholders need to be aware of them? Make sure everyone involved is clear on how changes will be prioritized.
• What questions are still unanswered? If you have unanswered questions, you should supplement your research with another method. For example, if your B version didn’t perform as well as expected in your recent A/B test, you may need to follow up with a usability test to find out why.

This is an excellent opportunity to combine qualitative and quantitative methods to arrive at a full-picture understanding. 

Quantitative research like surveys and A/B tests will help you understand trends around your target audience’s attitudes and behaviors and can help you predict future results. Qualitative research, like usability tests, will fill in the gaps by explaining what’s going on in the user’s head that causes them to take action or make a choice. 

Compare your quantitative and qualitative findings to get the big picture, bring your personas to life, and help your team empathize with the user.

Share your results

When it’s time to share your research with the teams implementing changes, presenting negative findings without stepping on toes or discrediting someone’s hard work can be tricky. Start by reminding everyone that you’re all working together to build the best possible experience for your users. Then present the findings objectively, with as many direct quotes or video clips from test participants as possible. Focus on the opportunity to create solutions for the issues rather than placing blame.

Of course, you’ll probably have quite a few positive findings, too! Please include them in your presentation or report to validate what’s going well. It’s just as important to understand what’s working as what’s not.

Finally, your findings can go beyond the design and research teams.

Share your results with:

• Customer-facing teams: To help them understand the customer journey
• The marketing team: To help them get inside the head of the buyer
• The executive team: To get buy-in for budget allocations and to keep them close to the actual user

Sharing your findings with other teams will also help you consolidate your research. You might find that different teams in your company are all doing their research through different channels but not sharing their findings. Create a shared space for your company’s research, and make sure all stakeholders have access to it. This can be a Google Drive folder, an Evernote notebook, or any other space allowing you to include members of multiple teams. This will prevent you from wasting time duplicating research that’s already been done.

With all of your research completed and shared, you can inspire your whole company to keep users at the heart of everything they do. By championing the user experience among your colleagues, you can build a user-centric culture at your company.