Usability Testing with Five Users: What’s Behind the Rule of Thumb

Picture a small product team working on “Pawplan,” an app that lets people book appointments at their vet. On the other side, practice staff manage the appointment calendar in a web application. Six weeks before launch, it is finally time to test. In the planning meeting, someone says the sentence that comes up in almost every meeting like this: “Nielsen says five users are enough.” Product management pushes back right away: “Five people? That doesn’t prove anything.” In the end, the team settles on one big test with twelve people, a mix of pet owners and practice staff, shortly before launch.

Both positions contain some truth, and in this case both miss the mark. The five-user rule is one of the best-known rules of thumb in UX design and also one of the most misunderstood. It comes from a mathematical model with clear assumptions. Once you know those assumptions, you know when five participants are exactly right and when they are nowhere near enough.

This article explains where the number comes from, what criticism and follow-up studies have shown and how to plan test rounds that actually improve your product. Pawplan serves as a running example throughout. It is made up, but typical.

Where the five comes from

A usability test is a method in which real people complete typical tasks with a product while you watch where they hesitate, make mistakes or give up. In its qualitative form, it is not about statistics. It is about finding problems and understanding them.

Researchers were already asking how many participants this takes in the early 1990s. In 1992, Robert Virzi published a paper in the journal Human Factors titled “Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?” His key finding: the first four to five participants uncover around 80 percent of the problems, and severe problems tend to show up early.

A year later, Jakob Nielsen and Thomas K. Landauer presented a model at the INTERCHI ’93 conference that is still cited today. They analyzed eleven studies and found that the discovery of usability problems can be described well as a random process, both in tests with users and in heuristic evaluations by experts. This leads to a simple formula for the share of problems found:

Share found = 1 − (1 − L)^n

Here, n is the number of participants and L is the probability that a single person uncovers a given problem. Nielsen later gave 31 percent as the typical value for L, averaged across the projects his team had studied.

What the math says

Plug L = 0.31 into the formula and you get the familiar curve that rises steeply at first and then flattens out:

• 1 participant: about 31 percent of problems
• 3 participants: about 67 percent
• 5 participants: about 84 percent
• 10 participants: about 98 percent
• 15 participants: almost 100 percent

That is where the claim comes from that five people find about 85 percent of the problems. Each additional person adds less that is new, because they mostly repeat what others have already shown you.

What Nielsen actually recommended in 2000

The rule became popular through Nielsen’s 2000 article “Why You Only Need to Test with 5 Users.” Read it today and you will notice it is far more nuanced than the rule of thumb it spawned.

Nielsen starts with a simple observation: zero users give zero insights. The very first person delivers almost a third of what there is to learn. After that, returns diminish. After the fifth user, Nielsen writes, you are wasting your time observing the same findings over and over.

The crucial part comes next. According to the curve, you need at least 15 users to find virtually all problems. But Nielsen explicitly does not recommend spending those 15 people on a single test. Three tests with five users each, with redesigns in between, are better. His reasoning: a second test shows whether the fixes worked, because there is no guarantee that they did. New designs can introduce new problems. And some deeper issues only become visible once the obvious ones have been cleaned up.

Nielsen adds two caveats right away. First, the formula only holds for comparable users who use the site in fairly similar ways. Second, you need more participants when there are several highly distinct user groups, such as children and parents. For two groups, he recommends three to four users per group. For three or more groups, three per group.

So five was never meant as a minimum size for a one-off check. It is the size of one round in an iterative process, for one homogeneous group.

What the rule of thumb does not say

The most common misunderstanding lies in the sentence “Five users find 85 percent of all problems.” Jeff Sauro and James Lewis of MeasuringU put it more precisely: five users are very likely to find the problems that affect roughly three in ten people. For less frequent problems, the math looks very different.

Take a problem that only one in ten people runs into. Using the same formula:

• With 5 participants, the chance of seeing it at least once is about 41 percent.
• With 10 participants, it is about 65 percent.
• For 85 percent certainty, you need around 18 participants.

A problem that affects only one in a hundred people goes unnoticed in about 95 percent of tests with five participants. The 31 percent is an average across many problems. Frequent problems show up fast, rare ones hardly at all.

On top of that, L is not a law of nature. Nielsen and Landauer themselves reported that the value varies widely between studies. Sauro and Lewis cite values between 0.03 and 0.60 from studies with large samples. For a complex application with many features, L can be much lower than for a simple checkout. And if you estimate L from the first two or three sessions of a test, you will typically overestimate it, because the problems nobody has encountered yet are missing from your data.

Criticism and follow-up studies

Since 2000, the rule has been hotly debated. The key contributions show where five reaches its limits.

Spool and Schroeder (2001): open tasks on large websites

In 2001, Jared Spool and Will Schroeder presented a short paper at the CHI conference with the pointed title “Testing web sites: five users is nowhere near enough.” Participants were free to browse up to four websites looking for CDs they wanted to buy, rather than working through fixed tasks. The authors concluded that these sites would have needed considerably more than five users to find 85 percent of the problems. Secondary sources report that the first five participants uncovered only about 35 percent of the problems and that later participants kept finding new ones.

The core of this critique: on large websites with open-ended tasks, people take very different paths, and the chance that two people hit the same problem drops.

Faulkner (2003): the luck of the draw

Laura Faulkner of the University of Texas tested the question empirically. She tested 60 people and randomly drew groups of different sizes from that data. The result shows how much chance matters with small groups: some groups of five found 99 percent of the problems, others only 55 percent. On average, groups of five did land at around 85 percent, which confirms Nielsen’s curve. But you cannot know in advance whether a particular test gives you a good group or a bad one. With ten users, the worst result rose to 80 percent. With twenty, it rose to 95 percent.

Faulkner’s conclusion: larger samples reduce the risk of missing important problems. That does not directly contradict Nielsen, but it shifts the focus from the average to the spread.

More objections: heterogeneity, tasks and analysis

In 2001, Alan Woolrych and Gilbert Cockton argued that problems do not affect people evenly, so a single average value for L can be misleading. In a 2010 meta-study, Wonil Hwang and Gavriel Salvendy arrived at a different rule of thumb: finding 80 percent of problems takes ten plus or minus two people. In 2012, Martin Schmettow of the University of Twente countered that even this number is too optimistic. Because problems vary in how visible they are and many studies remain incomplete, he argued, the formula systematically underestimates the sample size you need. Schmettow’s conclusion: the search for a magic number is doomed to fail, whether it is five, ten or 56.

Another finding concerns not the number of people but the tasks. In 2007, Gitte Lindgaard and Jarinee Chattratichart reanalyzed data from a comparative study in which nine experienced teams tested the same website independently. Although every team tested five or more users, the teams reported only 7 to 43 percent of the known problems. The results showed no reliable link to the number of users, but they did correlate with the number of tasks. Schmettow also points to Rolf Molich’s Comparative Usability Evaluation series, in which different professional teams testing the same product regularly reported markedly different results.

This leads to an uncomfortable truth: how many problems you find depends at least as much on what you test and how you analyze it as on how many people you invite.

When five participants are enough

Despite all the criticism, five is a good choice in many situations. Even Schmettow writes that most practitioners will keep relying on iterative, low-budget testing, where precise numbers are unreliable but also unnecessary. Five people work when three conditions come together.

You are testing qualitatively. The goal is to find problems and understand their causes, not to measure success rates or task times. If three out of five people get stuck at the same spot, you do not need statistics to know something is wrong there.

You are testing iteratively. After the test, the team revises the design and tests again. Whatever the first round misses gets a new chance in the second and third. And once the most frequent problems are fixed, they no longer block the view of the less frequent ones.

You are testing one homogeneous group. All participants have similar goals, similar prior knowledge and use the same part of the product. That way, the assumption that everyone runs into problems in similar ways is reasonably met.

Nielsen reaffirmed his recommendation in a 2012 article: five users for qualitative studies. He also mentions deviations in both directions. In very lean, agile projects, as few as two users per round can make sense. In consulting projects, his team often tested around eight users, partly because larger numbers are more convincing to executives.

When five participants are not enough

When you need numbers

As soon as you want to collect metrics, such as a success rate, a task time or a System Usability Scale score, different rules apply. With five people, any percentage is extremely uncertain. If four out of five complete a task, the actual success rate in your audience could be far lower or higher.

The Nielsen Norman Group now recommends 40 participants for quantitative studies in most cases. Raluca Budiu and Kate Moran derive this number from a sample-size calculation for a success rate, with a margin of error of 15 percentage points at 95 percent confidence. If you can accept more uncertainty, you can get by with fewer. Comparisons between two versions, for example in an A/B test, require their own calculations as well.

When there are several user groups

Five people per round applies to one group. Pawplan has at least two: pet owners who book appointments in the app, and practice staff who manage them in the calendar. These groups have different tasks, different devices and different prior knowledge. Lump them together and you end up testing each group with only part of your participants, learning too little about either.

Nielsen recommends three to four people per group in such cases and refined this in 2012: the more differently the groups behave, the closer you should get to five per group. Clear audience definitions, for example in the form of personas, help here.

When rare problems are costly

The math above shows it: problems that affect only a few people usually stay invisible in small tests. In a pet appointment app, that is annoying. In medication dosing, a bank transfer or an industrial control interface, a rare problem can have serious consequences. In those areas, small rounds are not enough as your only safeguard. You need larger samples, several methods and a systematic review of critical workflows.

When the product is large and the tasks are open

Spool and Schroeder showed that open tasks on large websites undermine the formula’s assumptions. For large products, though, Nielsen does not recommend putting more people into one test. He recommends several focused studies, each covering one area. Five people who all test the same checkout tell you more about that checkout than five people roaming freely across an entire website.

When you are using a different method

Five applies to qualitative usability tests. For other methods, Nielsen gives other numbers: at least 15 people per group for card sorting, and 39 people for stable heatmaps in eye tracking.

Planning iterative test rounds

Back to Pawplan. Here is the original plan, and here is what it looks like after a rethink:

• Before: One test with twelve people six weeks before launch, pet owners and practice staff mixed together. Afterwards, a long report with 40 findings, of which the team can only implement a fraction before launch. Nobody checks whether the fixes work.
• After: Three rounds with five pet owners each for the app, two weeks apart. In parallel, two rounds with five practice staff each for the calendar. Between rounds, the team implements the most important findings and checks them in the next round.

That adds up to 25 sessions instead of 12, but spread over eight weeks and each with a clear focus.

1. One group, one question per round

For each round, define who you are testing and which question you want to answer. For Pawplan, the first round’s question is: “Can pet owners find a suitable appointment, and do they understand what they have booked?” The tasks follow from that: book a vaccination for the cat, reschedule an appointment, find out what to bring to the visit.

Since Lindgaard and Chattratichart found the number of tasks to matter at least as much as the number of people, it pays to choose tasks carefully. Cover the most important workflows, not just the easiest path.

2. Recruit the right people

Five people only help if they are the right ones. For Pawplan, that means people who actually have a pet and take it to the vet regularly, not people from your own office. Plan for one backup participant, because cancellations happen. The glossary entry on user research covers how to set up studies like this.

3. Analyze quickly, prioritize, implement

Analyze each round promptly, ideally together with the team members who observed the sessions. Record how many people struggled with a problem, how severe the consequences were and where it occurred. The article How to Extract Valuable Feedback from Usability Tests explains how to turn observations into solid insights.

In Pawplan’s first round, three out of five people fail because the appointment type is called “Preventive care” while they are looking for “Vaccination.” Two people accidentally book at the wrong practice, because the practice selection only appears at the very end. Both issues are changed before the next round.

4. Check in the next round

The second round tests the revised version with five new people. Did the renaming help? Does the practice selection now appear in the right place? Round two reveals a problem nobody had noticed before: because booking is now faster, two people miss the note that they need to bring the vaccination record. This is exactly the effect Nielsen describes: some problems only become visible once the obvious ones are gone.

5. Know when a round is done

A practical signal: when the last two people in a round reveal hardly any new problems and mostly confirm known ones, the round has done its job. If the fourth and fifth person keep revealing new issues, that suggests the group is too mixed, the tasks are too open or the product still has fundamental problems. Then it is usually better to sharpen the setup than to keep testing regardless.

Limits and pitfalls

Percentages from small rounds. “80 percent of participants had problems” sounds precise, but with five participants it just means four people. In reports, write “four out of five” instead. It is more honest and keeps anyone from mistaking the number for a measurement.

Five people as proof of absence. A test with five people can show that a problem exists. It cannot show that there is none. If nobody failed at a certain point, all you know is that no frequent problem surfaced there.

Only one round. Testing with five people and then stopping uses the rule of thumb exactly the way Nielsen did not intend. The value of a small sample comes from repetition.

The wrong people. Five people from your own sales team are not five participants from your target audience. Their expectations and prior knowledge differ, and the formula assumes that participants represent your future users.

Testing as the only method. Usability tests find problems people have while completing tasks. They do not replace an expert evaluation, an analysis of usage data or an understanding of why people use a product in the first place. The strongest results come from several methods working together.

Conclusion

The five-user rule is neither a myth nor a law. It comes from a model that works well under certain assumptions: qualitative tests, a homogeneous group, problems that affect a meaningful share of people and, above all, several rounds with redesigns in between. Nielsen himself never recommended testing once with five people and then stopping. He recommended spreading the budget for 15 people across three rounds.

The criticism from Spool and Schroeder, Faulkner, Schmettow and others shows where those assumptions break down: in large products with open tasks, with distinct user groups, with rare but critical problems and wherever numbers are required. For the Pawplan team, this means no bigger test, but several small rounds, separated by user group. Five people are enough, as long as you keep going afterwards.

Checklist

• The goal of the test is clear: finding problems (qualitative) or measuring metrics (quantitative).
• Qualitative rounds plan for about five people per round and per group.
• For quantitative metrics, the sample is considerably larger and calculated, not guessed.
• Each clearly distinct user group is tested separately, with three to five people per group.
• Several rounds with redesigns in between are planned, not one big test.
• Each round has a clear question and tasks that cover the most important workflows.
• Participants belong to the real target audience.
• Findings are reported as “three out of five,” not as percentages.
• Critical, rare problems are covered with additional methods or larger samples.
• The next round checks whether the fixes from the previous one work.

Sources

• Jakob Nielsen and Thomas K. Landauer (1993): A mathematical model of the finding of usability problems. Proceedings of INTERCHI ’93, ACM, pp. 206 to 213
• Jakob Nielsen (2000): Why You Only Need to Test with 5 Users. Nielsen Norman Group
• Jakob Nielsen (2012): How Many Test Users in a Usability Study? Nielsen Norman Group
• Robert A. Virzi (1992): Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough? Human Factors, 34(4), pp. 457 to 468
• Jared Spool and Will Schroeder (2001): Testing web sites: five users is nowhere near enough. CHI ’01 Extended Abstracts, ACM, pp. 285 to 286
• Laura Faulkner (2003): Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), pp. 379 to 383
• Martin Schmettow (2012): Sample size in usability studies. Communications of the ACM, 55(4), pp. 64 to 70
• Gitte Lindgaard and Jarinee Chattratichart (2007): Usability testing: what have we overlooked? Proceedings of CHI 2007, ACM, pp. 1415 to 1424
• Kath Straub (2007): Why “how many users” is just the wrong question. Human Factors International
• Jeff Sauro, MeasuringU: A Brief History of the Magic Number 5 in Usability Testing
• Jeff Sauro and James R. Lewis (2020): Predicting the Number of Usability Problems Using Average Problem Occurrence. MeasuringU
• Raluca Budiu and Kate Moran (2021): How Many Participants for Quantitative Usability Studies: A Summary of Sample-Size Recommendations. Nielsen Norman Group