Why 5 UX Testers Are Almost Never Enough

There’s a pervasive myth in UX circles that you can get meaningful results by conducting user tests with only 5 participants. This is an appealing notion as it can be costly and time-consuming to conduct tests. However, the reality is that 5 users are almost never enough if you want trustworthy data. In this article, I’m going to explore where this myth came from and why it’s a flawed principle.

Who Started the Idea That You Only Need 5 Users?

There are many theories about user testing and you can find differing opinions depending on whom you ask. However, the idea that 5 users are enough was popularized by Jakob Nielsen. In his extremely influential article, Nielsen claims that the best approach to testing is to use no more than 5 users and to run numerous small tests. One of Nielsen’s conclusions is that it’s a waste of time to have more users because you’ll simply be seeing the same results repeated without learning anything new. Although Nielsen’s article was published in 2000, many UXers still take it as gospel. Unfortunately, there are several significant flaws in the 5-user theory. If you follow this idea without seeing its limitations, you may end up with incomplete or incorrect data.

Flaws and Limitations of Nielsen’s Theory

Here are the top reasons that you should carefully consider all the facts before following Nielsen’s advice about only needing 5 users.

It’s Based on UX From the 1990s

Nielsen’s article was published 20 years ago, which may as well be a century ago in the fast-paced world of UX. His findings are actually grounded in research and practices that date back even further, when websites were still a novelty. This is analogous to automakers today using the same methods as those used in Henry Ford’s first factory from the early 1900s. While in both UX and autos, some practices are enduring, there have also been many advances.

It Assumes Only a Single Audience Segment

If you are testing for more than one type of audience, 5 users would not be enough. The following are examples where you’d need to test distinct types of users.

• Testing users on different devices and/or operating systems.
• A website for an employment agency that has sections for both job seekers and businesses seeking employees.
• A business that has both B2C and B2B customers.
• A real estate company that has separate areas for buyers, sellers, and renters.

You could, of course, have the same 5 users take on different roles. However, it can get confusing for the participants if they have to keep switching roles, devices or operating systems. It’s more efficient to have distinct users for each segment. To his credit, Nielsen was aware of this limitation and stated that you need more users if you have multiple types of customers. However, not everyone is aware of this qualification of Nielsen’s principle.

It Doesn’t Apply to All Types of Testing

Jakob Nielsen understood that his model wasn’t applicable to all types of testing. Once again, this isn’t always acknowledged by those who quote him. In his 2012 article, How Many Test Users in a Usability Study, he listed a few instances in which more than 5 users are necessary.

• Quantitative Studies — If you need statistically significant numbers, you should test at least 20 users.
• Eyetracking — Nielsen recommends testing at least 39 users for stable heatmaps.
• Card Sorting — Use at least 15 testers.

Keep in mind, the above recommendations are from Nielsen himself. Thus, even if you accept his reasoning, you still shouldn’t assume that any UX test only requires 5 users.

Points to Keep in Mind For Usability Tests

When conducting usability tests, it’s important to consider your particular needs. Hard and fast rules, whether Nielsen’s 5-User rule or any other, are unlikely to apply universally. As noted, even Nielsen understood that there were exceptions and limitations to only having 5 users for tests. Here are some principles that can guide you when coming up with the ideal number of test subjects.

More Users Means More Reliable Data

Laurie Faulkner is one researcher who found evidence that revealed measurable benefits to increasing sample size. She compared tests using different numbers of users such as 5, 10, and 20. In one study, she found that 5 users sometimes identified 99% of problems but at other times only 55%. With 20 users, the lowest percentage of problems identified was 95%. Faulkner’s research is powerful evidence that more users yield more reliable results.

In some cases, there’s the additional factor that you need to show the results to a stakeholder such as an owner, manager, or senior executive. In such instances, apart from the objective reliability of the results, it’s more convincing when you have more data. For example, suppose you want to point out an issue with a website in which users are having difficulty logging in. It’s more convincing if you show that 16 out of 40 users have an issue as opposed to 2 out of 5.

When 5 Users is Sufficient

We’ve examined some reasons that you often need more than 5 users for reliable testing. However, this isn’t always the case. For certain tests, 5 users may be sufficient. For example, if you’re conducting early-stage exploratory research, you don’t need a large number of users. This includes:

• Testing the overall functionality and UX of a website or product.
• Identifying bugs or issues that affect usability.
• Seeking areas where UX can be improved.

During the early stages, you are looking for fundamental issues that impact UX. While it never hurts to have more users, it’s not essential.

Comparative or Competitive Benchmarking Requires More Users

These are tests that measure factors such as:

• How much time users spend on a task.
• The path taken by users to accomplish a task.
• Errors made by users.
• How often users need outside help to complete a task.

For these and similar tests, you need a certain number of users to get accurate and actionable data.

Find the Right Number of Users for Each Test

The idea that you only need 5 users is an example of a widely-accepted doctrine that isn’t questioned enough. It’s a case of people repeating something so often that many just assume it must be right. Considering that it’s based on research dating back to the early days of the internet, it certainly deserves closer scrutiny. While there are instances in which 5 users is sufficient, most UX testing is better served by having more testers.

The real lesson here is that you shouldn’t make a decision on the number of testers based on a general principle. You have to consider your particular situation and what you’re testing. You also need to consider who will see your data, such as your manager or client. There’s little doubt that more users yield more trustworthy results.