Using rating scale questions in your UX research

In a previous post, Streamline your UX research with customer experience analytics, we discussed the overall benefits and guidelines that UserTesting’s Research team keeps in mind as they incorporate survey-style questions into their studies. One tool they really like to use—and recommend you do, too—are rating scale questions.

The good news is, that it’s not as hard to design effective rating scale questions as you might think. Keep reading to learn how to get the data-backed support for the qualitative findings you need.

What is a rating scale?

Rating scale questions gather answers that represent a range of choices along with particular themes, like satisfaction level, how likely a respondent is to do something, and the extent to which a respondent agrees with a particular statement.

Here are a few examples:

• How difficult (1) or easy (5) was it to log in to the app using Facebook? (1 = Very difficult, 5 = Very easy)
• How unlikely (1) or likely (5) would you be to recommend this game to a friend? (1 = Not at all likely, 5 = Extremely likely)
• How cluttered (1) or organized (5)  do you consider the home page? (1 = Cluttered, 5 = Organized)
• How untrustworthy (1) or trustworthy (5) do you consider the site? (1 = Not at all trustworthy, 5 = Entirely trustworthy)
• How disinterested (1) or interested (5) are you in purchasing boosters for this game? (1 =  Not at all interested | 5 = Extremely interested)
• Please rate your agreement with the following statement: "I understand what I can do on this site." (1 = Strongly disagree | 5 = Strongly agree)

Why use a rating scale? 

Rating scale questions help you quantify abstract or intangible concepts with approximate answers constrained to a single, logical value set (e.g., very likely to not at all likely, 1-7, 1-5). 

As the most utilized type of metric question, rating scale questions:

• Give you the easiest indications of user highs and lows
• Provide information about respondents’ attitudes, behavior, and responses to your product or service
• Result in quantitative data you can use to create charts and graphs that visually convey your survey results

Types of rating scales

There are four main types of rating scales.

Ordinal scales

Ordinal scales deliver answer sets that occur in a logical, systematic order and have a relational link. For example:

• Ask respondents to use a scale of 0-10,  where 9-10 identifies the respondent as a “promoter”  to denote how likely they are to refer a company/product/service to someone else (i.e., a Net Promoter Score).

Likert scales

Likert scales assess the degree to which a respondent agrees or disagrees about a given statement. For example:

• Ask respondents to use a 1-5 scale where 1 represents “I strongly disagree” and 5 represents “I strongly agree” to communicate how strongly they agree or disagree with a particular statement.

Semantic differential scales

Semantic differential scales gauge emotional attitudes toward a topic by asking respondents to rate a product, company, service, etc. within the frames of a multi-point rating option representing opposite adjectives at each end. For example:

• Ask respondents to use a multipoint spectrum where one end represents “Extremely dissatisfied” and the other end represents “Extremely satisfied” to indicate their beliefs about a product/company/etc.

Interval scales

Interval scales feature answer sets where each interval represents a deeper meaning, not just an ordered grouping. For example:

• Ask respondents to use a 1-5 ordered list of numbers where 1 represents how “Least likely” and 5 represents how “Most likely” they are to complete a specific action.

How to make the most of your rating scale questions

Now that you understand the basics, here are some tips to help you get the most out of the rating scale questions you write.

1.  Clarify the endpoints of your rating scale

Rating scales are highly adaptable and can be used to measure many different things—but only if the people taking your test understand what you’re asking them to measure. So make sure your rating scale involves two ends of a spectrum (which you effectively communicate) and assign clear labels to each one. 

2. Include both sides of the rating scale in your question, as well as in the endpoint fields.

If you ask users how easy it was to accomplish something, their inclination is to say it was easy—even if they struggled—because people are naturally eager to please. You can avoid introducing bias by making sure you include both endpoints in your question (e.g., the good/the bad, how hard/how easy, etc.).

Pro tip: If you include both sides of the scale in the body of your question, you’ll also reinforce the endpoints you’ve defined and lower the chance a user will misread your question.

3. 1 should always be the “pain” point of the rating scale, and 5 should always be the “positive” point

We tend to consider low numbers to be bad and high numbers to be good. Looked at another way: we automatically associate higher numbers with good news and lower numbers with bad news. So do yourself — and your users — a favor, and always label 1 as the “pain” point, and 5 as the “positive” point.

Pro tip: If you set up a rating scale question and realize the two endpoints are neutral and/or the positive point is right in the middle, consider turning it into a multiple-choice question. Instead of asking users to rate something on a scale from (1) “Too small” to (5) “Too big,” for example, change the question to a multiple-choice format featuring three answer options — “Too small,” “Just right,” and “Too big.”

4. Use the optimal number of points on your rating scale.

You want respondents to differentiate their answers as much as possible, but you also don’t want to provide so many points that your rating scale becomes confusing or unreliable. It’s a delicate balance. So here are two tried-and-true guidelines that can help.

For ideas that range from positive to negative, use a 1-7 point scale that includes a middle or neutral point.

And for ideas that range from zero to positive, use a 1-5 point scale. 

5. Measure one element or action at a time with your rating scale

Imagine you easily download a new app and then spend seven extremely frustrating minutes trying to figure out how to create a new account. 

Then you’re confronted with the following question:

By asking about two different elements—downloading the app and setting up an account—this question is trying to measure two distinct elements. And it will yield invalid, inconclusive, and misleading results So be sure to write separate, individual rating scale questions for each task your users perform.

Pro tip: If you’re worried about users mistaking the second question as a duplicate, you can always capitalize the task you’re asking users to rate, for additional emphasis:

6. Gain deeper context by allowing additional information in your rating scale.

Allowing respondents to expand upon their rating answers can help you understand why they answered the way they did. It can also alert you to problems and opportunities with your offering. So include a free-text answer box underneath each rating scale question where respondents can input any additional information they want to share about why they chose their answer.