UX testing methods

What are the different types of usability testing methods? 

It’s one of the most crucial questions that will shape your usability testing project: whether to conduct moderated or unmoderated usability testing. Let's look at both to understand the advantages and disadvantages of each. 

Unmoderated vs. moderated research: What's the difference? 

At a high level, the difference between moderated and unmoderated is whether a researcher (the moderator) oversees the test or the test participant is left to carry out the task without any real-time supervision.  

Of course, people often ask if moderated usability testing is better than unmoderated testing and vice versa. The truth is that both forms of testing are valuable in different contexts.

Moderated usability testing

In a moderated usability test, a researcher will be on hand to facilitate the test in real time. If the test is conducted remotely, the researcher will observe the participant via their testing software tool in real-time. 

During the test, the moderator will ask the participants questions, guiding them through the session to understand their behavior and discover potential usability issues. 

Typically, you'll create a test script ahead of the session as a form of structure—but the perk of being in a live setting is that you can ask questions on the fly. For example, if a user interacts with your solution unexpectedly, you can find out why there and then. 

Another benefit of moderated testing is inviting stakeholders and teammates to observe the session anonymously. When stakeholders watch a test first-hand, they often start to grasp the benefits of user research and are more likely to champion further projects. 

When should I use moderated usability testing?

Moderated studies have beneficial use cases throughout the design and product lifecycle. As a general rule, moderated testing is the ideal option whenever you need to converse with users or gain in-depth feedback. 

In particular, we advocate moderated testing for: 

• Validating high-level concepts: Whether it's an early-stage prototype or just a one-page concept, you can use moderated studies to validate your ideas with people and gauge whether they're worth pursuing further. 
• Testing complex products: Some solutions are too difficult to explain through written instructions. If you feel like you want to be "in the room" with your user, moderated testing is your best bet. 
• Formulating and refining your research hypotheses: One of the most important phases of research involves developing a hypothesis about how you expect users to interact with your product — based on historical data and personas. Moderated research allows you to refine these hypotheses granularly. In the long run, this helps you create research projects that are clearly defined and rock solid.
• Creating UX advocates: Inviting stakeholders to observe moderated sessions is a surefire way to showcase the value of usability testing. You'll be amazed at the levels of empathy your stakeholders build.  
• Testing logged-in environments: Most websites and applications require users to create a dedicated log-in account, making it challenging to test logged-in settings. Moderated studies can help you to overcome this difficulty. For example, you can have a moderator log in and then pass control over to the tester. 

Pros of moderated testing

• Great for conducting user interviews and gathering rich customer journey feedback.
• It enables you to understand your participants' thoughts and feelings about your solution—something that isn't achievable with qualitative research methods.
• Video-led discussions allow you to pick up on non-verbal cues like body language and facial expressions, which can generate further insights.
• You can support participants through complex tasks.

Cons of moderated testing

• Testing can be lengthy and time-consuming 
• You need to be wary of asking leading questions, which could skew your results 
• Sample sizes are inherently smaller in moderated testing, so there's often less confidence in broad insights
• Time-intensive and planning-heavy, this form of testing tends to be expensive 

Unmoderated usability testing

With unmoderated usability testing, no researcher can observe the test in a live setting. The only person in the session is the participant, who works through a pre-defined list of tasks and questions you've assigned. 

A great benefit of unmoderated testing is that the user can perform their tasks at any time of their choosing, freeing up their time to focus on other projects as your research project effectively runs in the background. 

Because you don't have the opportunity to interact with your user, you'll need to be crystal clear in your instructions so that the script, scenarios, and tasks feel logical without any further clarification.

To get your results from an unmoderated test, you'll lean heavily on your usability testing tools, which we discussed in the last chapter. 

If you're running a qualitative study, you'll use recording software to view the session later and analyze the user's thoughts and behaviors. If you're taking the quantitative approach, your tool should automatically collect and analyze the data to generate numerically based insights. 

When should I use unmoderated usability testing?

If you want quick insights from a wide range of participants, unmoderated usability testing is the best method. Generally, we advise this form of testing in cases where you need specific answers, want feedback on your solution ASAP, or want to see a participant interacting with your product in their natural environment. 

Unmoderated is ideal for: 

• Meeting Agile demands: Agile is the name of the game today. Engineers and product managers often wait for your UX insights, so you need to deliver them stat. In this fast-paced environment, unmoderated studies facilitate the need for speed. 
• Scaling your research: Statistical significance is gained through larger sample sizes. Because unmoderated testing leaves users to their own devices (literally), you can better scale your project, potentially including hundreds of people.
• Using quantitative data to cement qualitative insights: Qualitative and quantitative research don't have to be mutually exclusive—they can work great together. For example, you could run a qualitative study and discover that a handful of users aren't satisfied with a design feature. You can then use a large-scale quantitative test to prove or disprove this insight. 
• Democratizing UX research: The demand for UX research is higher than ever. You only have 24 hours daily, so you and your team can't run every research project requested. People from other departments and roles can engage in research with unmoderated research. With the right platform and some training, you can empower your colleagues to run their own unmoderated tests at speed and scale. 
• Finding your tribe: Unmoderated research unlocks the potential for you to gain feedback from participants anywhere in the world. Whether you need a geographically dispersed sample size or need to reach a specific group of people with your product, unmoderated research has your back. 

Pros of unmoderated testing

• Without a moderator, these tests are inexpensive to run at scale. 
• The statistical significance of your sample size improves confidence in your insights.
• Access participants from all over the world.
• Empower non-UX professionals to conduct their own research projects.

Cons of unmoderated testing

• You may need to conduct various rounds of testing to uncover the insights you seek.
• Statistical insights give you an idea of what users are doing—but you won't understand why without qualitative research.
• You must be careful about your instructions—confusing directions or a complex product can undermine your research project, leading to poor results.

As you can see, each strategy has its use cases—so it's important to figure out your priorities and consider blending multiple methods before you begin your research project.