Measuring UX: Task Success, Time on Task, SUS and Conversion Explained

Picture a small team that has just redesigned the online booking flow for a physiotherapy practice. The new version looks cleaner, and everyone on the team agrees: this is much better. Then the practice manager asks, “How can we tell?” And suddenly the room goes quiet.

That is exactly where this article comes in. It shows you how to put user experience into numbers without a degree in statistics. You will get to know the four metrics that come up most often in practice: task success rate, time on task, the System Usability Scale (SUS) and conversion rate. And you will learn when each of them helps and when it is more likely to mislead you.

The booking example runs through the whole article. All of its numbers are made up and only serve as an illustration.

Why measure UX at all?

User experience (UX) describes how people experience a product, from the first impression to the finished task. Much of it shows up in observation: someone hesitates, hunts for a button or sighs at a form. Those impressions are valuable, but they are hard to compare. Was the new version really easier, or did the participants just have a better day this time?

Metrics make change visible. They help you with three things:

• Comparing: old versus new version, your product versus an average, or today versus three months ago.
• Convincing: a sentence like “before, two thirds managed to book; now almost everyone does” often convinces management faster than ten interview quotes.
• Prioritizing: if one task performs clearly worse than the others, you know where to look first.

There is one distinction you will run into a lot in the literature. Qualitative methods ask why: where do people fail, and what are they thinking? Quantitative methods ask how much: how many fail, how long does it take, how satisfied are they? The Nielsen Norman Group (NN/g), one of the best-known UX research and consulting firms, describes qualitative studies as mostly formative (they help you shape the design) and quantitative studies as mostly summative (they take stock). The two complement each other. Numbers tell you that something is off. Observation tells you why.

The three sides of usability

Before looking at individual metrics, it is worth asking what you actually want to measure. The international standard ISO 9241-11 offers a useful starting point. It defines usability as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

That sentence contains three questions you can translate directly into metrics:

• Effectiveness: do people achieve what they set out to do? Task success rate measures this.
• Efficiency: how much effort does it take? Time on task is one way to measure this.
• Satisfaction: how do people experience using it? A questionnaire such as the SUS measures this.

Conversion rate sits a little outside this grid. It does not measure experience but a business outcome that depends, among other things, on good UX. More on that later.

Task success rate: do people complete their task?

Task success rate, also called completion rate, is the simplest and often the most important metric. It tells you what percentage of participants complete a task successfully. It is usually measured in a usability test: people from your target audience get realistic tasks and try to solve them with the product while you observe.

How to measure it

Jeff Sauro of MeasuringU, a research firm that specializes in UX measurement, recommends simple coding: 1 for success, 0 for failure. The task success rate is then the number of successes divided by the number of attempts.

The crucial step happens before the test, though: you define what counts as success. For the booking flow, it could look like this:

Task: “Book a physiotherapy appointment for next week, in the afternoon.”

Success: the booking confirmation appears on screen with an appointment after 12 noon, in the right week and for the right type of treatment.

Without this definition, arguments start afterwards. Does it count as success if someone books a morning slot? If someone only finishes after a hint from the facilitator? Decide these cases beforehand and stick to your decision. As a rule, anyone who needed help did not complete the task on their own.

An example

In the example, the team tests both versions of the booking flow unmoderated, meaning without a facilitator, with 40 people per version. With the old version, 26 out of 40 manage to book. That is a success rate of 65 percent. With the new version it is 35 out of 40, just under 88 percent.

So is 65 percent bad? One reference point: Sauro analyzed 1,189 tasks from 115 usability tests. The average success rate was 78 percent. The old version falls below that, the new one above. Benchmarks like this are only a rough guide, though. A simple task such as “find the opening hours” should score much higher than a complex task in tax software.

Be careful with small groups

With only a few participants, percentages are deceptive. Sauro gives an example: if 10 out of 10 people complete a task, you can only say with 95 percent confidence that somewhere between 75 and 100 percent of your whole target audience would succeed. That range is called a confidence interval. The more people you test, the narrower it gets.

That is why NN/g draws a clear line: for qualitative tests, where the goal is to find problems, five people are often enough. For quantitative studies, according to NN/g, confidence intervals usually only become narrow enough to be meaningful from around 40 participants. So if you test with five people, say “four out of five managed it” rather than “80 percent success rate.” It is more honest and does not set false expectations.

Time on task: how much effort does it take?

Time on task measures how long people need to complete a task. It is a measure of efficiency. A booking that takes five minutes is annoying even if it works in the end.

How to measure it

According to Sauro, timing starts when the participant has finished reading the task and ends when they have completed all the steps. Tools for unmoderated testing usually record this automatically. For moderated tests, the recording is often enough to read off the times later.

Keep two things in mind:

• Separate success from failure. Someone who gives up after eight minutes was not slow, they were unsuccessful. Analyze the times of successful attempts separately, or you end up mixing two different questions.
• Keep conditions the same. If participants think aloud while working, that can slow them down. Only compare times between tests that are set up the same way.

Why the regular average is misleading here

Task times have a quirk: most people are similarly fast, but a few take much longer. Statisticians call such a distribution skewed. Those outliers drag the regular average, the arithmetic mean, upward.

For small groups of up to about 25 people, Sauro therefore recommends the geometric mean. It is an average that gives large outliers less weight. In Excel or Google Sheets, you calculate it with the GEOMEAN function. For larger groups, the median also gives a good picture. That is the value in the middle when you sort all times by size.

In the example, the geometric mean of successful attempts is 3 minutes 40 seconds with the old version and 2 minutes 10 seconds with the new one. That is a clear difference. What the time does not tell you, however, is where exactly people got stuck in the old version. For that, you need the recordings or notes from the test.

Satisfaction: how do people experience using the product?

Success and time describe what people do. They do not tell you whether people feel confident, annoyed or overwhelmed while doing it. That is what standardized questionnaires are for. Standardized means the questions are fixed and have been validated in many studies. This lets you compare your results with those of other products.

The System Usability Scale (SUS)

The best-known questionnaire is the System Usability Scale, or SUS. John Brooke developed it in 1986. It consists of ten statements, such as “I found the system unnecessarily complex,” which participants agree or disagree with on a scale from 1 to 5. The result is a single score between 0 and 100. The glossary entry explains step by step how to calculate it.

If you are just starting out, one point matters most: a SUS score is not a percentage. Sauro analyzed around 500 studies, and the average is 68. A score of 68 is therefore average, even if it sounds like “two thirds good.” According to Sauro, a product needs a score of 80.3 or higher to rank among the top 10 percent. Whenever you share SUS scores with others, always explain how they compare.

In the example, the old booking flow gets a SUS score of 61 and the new one 76. So the old version was below average, and the new one is clearly above it, though not yet in the top group.

Participants fill in the SUS at the end of a session, after they have worked through all the tasks. It therefore rates the product as a whole. According to Sauro, the questionnaire’s reliability does not depend on the number of participants, so you can use it in small tests as well. The fewer people fill it in, however, the less precisely the average estimates the score for your entire target audience.

The Single Ease Question (SEQ)

If you want to know which individual task feels difficult, the Single Ease Question, or SEQ, helps. It is one single question asked right after each task: “Overall, how difficult or easy was the task to complete?” People answer on a scale from 1 (very difficult) to 7 (very easy). According to MeasuringU, the average across more than 400 tasks is about 5.5.

The SEQ is quick, barely interrupts the test and shows you where something feels like hard work, even if the task was completed in the end.

Conversion: what happens in real life?

All the metrics so far come from tests with selected participants. Conversion, on the other hand, measures what happens in real use. A conversion is a desired action, such as a purchase, a sign-up or an appointment booking. The conversion rate is the share of visits that lead to that action.

The math is simple: conversions divided by visits, times 100. If, in the example, 2,000 people a month start the online booking and 1,300 of them complete it, the conversion rate of this flow is 65 percent.

NN/g distinguishes between two levels here. Macro conversions are the actual goals, in the example the completed booking. Micro conversions are smaller steps along the way, such as choosing a treatment type or an available slot. If you look at the micro conversions step by step, you can see where most people drop out.

What conversion rate cannot do

Conversion rate has one big advantage: it is based on the real behavior of a large number of people. But it also has two weaknesses that are easy to underestimate when you are starting out.

First, it does not depend on UX alone. Prices, seasons, ad campaigns or simply who ends up on the website influence it just as much. If the practice runs an ad in January that attracts lots of curious visitors, the conversion rate may drop even though the booking flow has improved. The most reliable way to check whether a design change really works is an A/B test, in which two randomly split groups see different versions at the same time.

Second, conversion rate says nothing about how people felt along the way. A booking that only succeeds after a lot of frustration counts just as much as an effortless one. And anyone who pushes the rate up with hidden costs or pre-selected add-ons gains numbers in the short term and loses trust in the long term.

Which metric helps when?

No metric is the right one on its own. It depends on your question. A few typical situations:

• You want to find problems in a draft or prototype. What you need most here is observation from a small qualitative test. The SEQ after each task is a good addition because it shows which task feels hardest. With five people, it is better to leave out percentages.
• You want to compare two versions before one goes live. Task success rate, time on task and SUS help here, with enough participants per version. This is the situation in the example.
• You want to know how your product is doing overall. The SUS is a good fit because published benchmarks exist for it. If you repeat the measurement regularly, say once a year, you also see how things develop.
• Your product is live and you want to track changes in everyday use. That is what conversion rate and micro conversions from your analytics tool are for, ideally combined with A/B tests for individual changes.

A framework for choosing: HEART

If you do not know where to start, a model published by a team at Google in 2010 can help. In it, Kerry Rodden, Hilary Hutchinson and Xin Fu describe five categories whose initials spell HEART:

• Happiness: attitudes and satisfaction, for example measured with a questionnaire.
• Engagement: how intensively or how often people use the product.
• Adoption: how many people start using it within a given period.
• Retention: how many are still using it after a while.
• Task success: success rate, time on task and errors.

It comes with a simple three-step process called Goals, Signals, Metrics. First you define the goal. Then you think about which behavior or attitude would show you that you are succeeding. Only at the end do you choose the specific metric. For the booking flow, that could mean: the goal is that people find a suitable appointment without calling the practice. A signal would be fewer abandoned bookings. The metric is the weekly conversion rate of the booking flow.

The authors stress that not every product needs all five categories. For software that employees have to use for their job anyway, engagement is not very meaningful, for example. The paper also warns against page views as a UX metric: more views can mean that something is popular, but just as well that people are wandering around a confusing interface.

Common mistakes when measuring

A few pitfalls are almost bound to come up when you start measuring UX:

• A number without comparison. “Our success rate is 72 percent” says little on its own. Only comparing it with an earlier version, another task or a benchmark makes the number meaningful.
• Percentages from tiny groups. Three out of five is not “60 percent” of your audience. Reporting small tests in percentages promises a precision that does not exist.
• Defining success after the fact. Then every borderline case turns into a debate, and in the end you measure what you want to see.
• The metric becomes the goal. When a team only optimizes for conversion rate, it quickly ends up with tricks that help the number and hurt people.
• Measuring instead of understanding. Numbers show you that something is wrong. You only find out what to change when you look closely. The article How to Extract Valuable Feedback from Usability Tests shows how to turn test observations into useful insights.

The example at a glance

Back to the physiotherapy practice. Instead of “this is much better,” the team can now show the practice manager the following:

• Task success rate: before, 26 out of 40 people (65 percent); after, 35 out of 40 (just under 88 percent).
• Time on task: before, 3 minutes 40 seconds; after, 2 minutes 10 seconds (geometric mean of successful attempts).
• SUS: before, 61 (below the average of 68); after, 76 (above it).

After the new version launches, the team also tracks the conversion rate of the booking flow. And it is already planning the next step: the recordings show that some people cannot find the treatment they are looking for because they expect a different label in the list. No metric alone would have revealed that insight.

Conclusion

Measuring UX is not rocket science. To get started, four metrics are enough: task success rate tells you whether people complete their task. Time on task shows how much effort it takes. SUS and SEQ reveal how people experience using the product. And conversion rate shows what happens in real use.

More important than any single number is the question you want to answer. Decide beforehand what you want to know, what counts as success and what you will compare against. And keep in mind that numbers point you toward problems, not toward solutions. You find those by watching people use your product.

Checklist

• The question is clear: finding problems, comparing versions, assessing the current state or tracking live behavior?
• For every task, what counts as success is defined in advance.
• The group size fits the method: a few people for qualitative tests, around 40 for reliable numbers.
• Times are analyzed separately for successful attempts, using the geometric mean or median.
• The SUS is used with as few changes as possible at the end of the session, the SEQ right after each task.
• SUS scores are put in context with the average of 68 and not read as percentages.
• Conversion changes are checked with an A/B test where possible, not just with a before-and-after comparison.
• Every number is backed up with observations that explain why it is what it is.

Sources

• ISO 9241-11:2018: Ergonomics of human-system interaction, Part 11: Usability: Definitions and concepts
• Jeff Sauro: What Is a Good Task-Completion Rate? MeasuringU, 2011
• Jeff Sauro: 10 Essential Usability Metrics. MeasuringU, 2011
• Jeff Sauro: Average Task Times in Usability Tests: What to Report? MeasuringU, 2010
• Jeff Sauro: Measuring Usability with the System Usability Scale (SUS). MeasuringU, 2011
• Jeff Sauro: 10 Things to Know About the Single Ease Question (SEQ). MeasuringU, 2012
• John Brooke: SUS: A Retrospective. Journal of Usability Studies, 2013
• Raluca Budiu: Why 5 Participants Are Okay in a Qualitative Study, but Not in a Quantitative One. Nielsen Norman Group, 2021
• Kathryn Whitenton: Macro and Microconversions as Metrics in Analytics. Nielsen Norman Group, 2019
• Kerry Rodden, Hilary Hutchinson, Xin Fu: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications. CHI 2010