Usability test questions: does wording matter?

Interaction Designers have a tough job. Designing is hard work! It requires the ability to be simultaneously creative and analytical – not an easy thing. When I changed jobs from being an Interaction Designer to being a UX Researcher, I initially assumed I would escape the rigors of the design role. However, I quickly realized that to be a good researcher, I had to have good research design skills. To get good research outcomes, you need to design effective research studies.

I also realized that, just like in interaction design, there are multiple ways to design in order to achieve a given objective. Each design option has a set of tradeoffs – each has its own strengths and weaknesses. Indeed, I have come to the conclusion that there is no such thing as a perfect design. But even so, a critical requirement of an effective research design is that it addresses and manages potential bias that can leak into a study and potentially skew or pollute research findings.

It’s important for UX researchers to be sensitive to the many ways that bias can affect a study, and to the extent possible, control for it. A key aspect of research design is how we craft the questions asked of research participants. It turns out that how we ask a question has critical implications for how a participant responds, and consequently, on research outcomes.

To see just how important this is, let’s consider a research study where a number of individuals were asked to determine how a judge should rule in a child custody case. After reviewing a list of characteristics about each parent, some individuals were asked which parent should get sole custody, while others were asked which parent should not get sole custody. Would simply asking the question in either a positive or negative way affect people’s response? After all, both questions are just different ways of asking essentially the same thing.

What they found is that the decision to either award or deny custody was indeed influenced by the way the question was asked. When people were asked which parent should get sole custody, they quite naturally focused on and compared the positive characteristics of each parent. When they were asked which parent should not get sole custody, they automatically compared the negative characteristics of each parent.

We can see that the wording of the question actually drove people’s thought process and what they focused on in the decision problem. This is huge! It means that the actual wording of the question has the power to drive attention and focus, and therefore, decision outcomes.

Sometimes as UX researchers, we get into the groove of only asking certain types of questions –
• What did you like about…?
• What would make this better?
• Describe your optimal experience….

Note that all of these questions are framed in a positive way. But what would happen if we asked the question differently?
• What don’t you like about…?
• What would make you stop using….?
• What would happen if the [thing you’re testing] didn’t have this feature/functionality….?

Sometimes the negative version of a question opens the opportunity to get insights from participants that you wouldn’t get otherwise. And asking both the positive and negative version of the question can result in a more holistic understanding of the user experience.

Another way to glean insights is to consider how broadly or narrowly you frame the question. Often, when you take a step back and ask a ‘broader’ question, you open the door to obtaining answers to questions you hadn’t thought to ask. Asking a broader question also helps control for assumptions that can affect and bias your interpretation of what you observe.

Let’s take a closer look at the questions above. Question #1 makes the assumption that the participant actually notices the differences. But what if s/he doesn’t? Now the participant feels uncomfortable and needs to ‘save face.’ Always remember that the participant will supply an answer to your question. That’s what s/he’s getting compensated for, after all. But, the answer you get may not reflect the reality of the situation.

The narrow wording of question #2 leads the participant to focus only on frequency of use, whereas the revised version allows the participant to answer in a way that allows you to see what’s most important to him. The wording of question #3 leads participants into a mindset of ‘ranking,’ which is a very different task and thought process than answering a broader question about which of the options is a better approach.

It’s worth taking the time to think through various ways of wording your research questions to appreciate how the wording itself may affect participants’ thought process, and consequently how they respond. Truly, research design is both an art and a science – and that’s what makes it so fascinating.

This is a post by Colleen Roller.  Colleen is forever fascinated with the workings of the human mind, and with the art and science of designing for it. She has written extensively on this topic, authoring columns for UXmatters and UX Magazine. On the personal side, Colleen is a classically trained musician and enjoys performing on alto and soprano recorder. She also designs jewelry, and you can find her necklaces in a popular shop in West Concord, MA.