Jakob Nielsen's Alertbox, February 22, 2010:

Progress in Usability: Fast or Slow?

Summary:
Over the past decade, usability improved by 6% per year. This is a faster rate than most other fields, but much slower than technology advances might have predicted.

The good news? We're making progress in usability. Websites and other user interfaces get better every year, and the improvements accumulate across the decades, making current quality levels substantially higher than those of the early days of the Web or the personal computer.

Even better: Usability progress is faster than most other forms of human progress.

The bad news? Usability advances at a much slower rate of progress than other areas of computing.

Even worse: it'll take us 74 years to reach acceptable user experience quality.

Usability Improvement Rates

I typically like to report success rates, because they're the simplest usability metric and very easy to understand: can people use the design or not? However, for this article, I'll consider the failure rate, which is simply the reverse of the success rate.

For example, a website success rate of 70% means that users are able to accomplish 70% of the tasks they attempt on that site. This, of course, means that they fail 30% of the time, which is the failure rate.

I use failure as the metric in my current analysis because it allows a more straightforward comparison with other fields of quality.

During the last decade, we've collected formal usability metrics for 262 websites. In 2000, the average failure rate was 39%; in 2010, the average failure rate is 22%.

That's what I mean by "substantial advance" over time. In just a decade, we've almost doubled Web usability. This improvement compares well with two other views of the big picture:

• Website conversion rates also approximately doubled, from an average of about 1% to about 2% across sites. (Some are better, of course, but some are worse.)
• Research on usability ROI shows that attention to usability approximately doubles desired business metrics.

(Our latest study in 2010 did slightly better than indicated above, but I have excluded those 15 sites from the present analysis, because this study was conducted for our "Big and Famous Sites" project. This data set is thus deliberately biased because we wanted to tease out the usability lessons from well-designed sites. So, for example, we tested bbc.co.uk because we know there are several good aspects to BBC's user experience. It's nice to see this confirmed by a 17% failure rate [which is a good score, sadly], but BBC's performance is not representative for sites with more average usability budgets.)

We can also analyze Web usability's long-term improvement in two other ways:

• The rate of improvement is 6% per year. By this, I don't mean that failure rates drop by six percentage points each year, but rather that each year's failure level is 1.06 times smaller than the previous year's level. Given the magic of compounding interest, improving that much in each of 10 years produces a cumulative improvement of 77%, which matches the relative failure-rate gain from 39 to 22.
• A six-sigma analysis of usability shows that, in 2000, we were at a 1.8 sigma level — which was horrible compared to any other field of quality assurance. Now, in 2010, we're at 2.3 sigma. With a mere 0.5 gain in sigma level for the decade, we still have far to go before we reach level 6.

This six-sigma analysis shows how bad usability is relative to industrial quality levels. At a rate of half a sigma per decade, it will take 74 years to reach six-sigma quality. (The only good news? This ensures lifetime employment for usability professionals.)

Progress in Other Fields

Usability improves at a rate of 6% per year. How does that compare with progress in other areas?

Stephen Moore and Julian L. Simon wrote a book called It's Getting Better All the Time that summarizes improvements in 100 different fields across the 20th century. Following are some of their statistics (all from the United States):

• Infant mortality: 100 per 1,000 births in 1915, dropping to 9 in 1998. Improvement rate: 3% per year.
• Children without dental cavities: 26% in 1971, increasing to 55% in 1988. Improvement rate: 5% per year.
• People showering or bathing daily during winter: 29% in 1950, increasing to 75% in 1999. Improvement rate: 2% per year.
• Time worked to buy a chicken (at an average worker's salary): 2 hours in 1920, dropping to 15 minutes in 1999. Improvement rate: 3% per year.
• S&P stock market index: 6 in 1900, growing to 1,400 by 1999. Improvement rate: 6% per year. (As it turns out, 1999 was a bubble year, and the S&P later dropped, but for the sake of the long-term analysis, I'm sticking with the book's 20th century data.)
• Toy sales: $2 billion in 1921 (adjusted to 1998 dollars), growing to $45 billion in 1998. Improvement rate: 4% per year.
• Farm productivity, sacks of onions per acre: 200 in 1950, increasing to 800 in 1999. Improvement rate: 3% per year.
• Deaths caused by Chicago heat waves: 10,000 in 1901, dropping to 300 in 1995. Improvement rate: 4% per year.
• Airplane speed: 37 miles per hour in 1905 (Wright brothers Flyer III), increasing to 2,070 miles per hour in 1965 (Lockheed YF-12A). Improvement rate: 7% per year.

I could go on (the book has 100 datasets), but the conclusion is clear: Human progress happens at 4% per year, averaged across many fields, ranging from 2% to 7%.

Suddenly, usability's progress rate of 6% per year doesn't look so bad. We're doing better than most other fields.

Why do we expect (even) better of usability? Two reasons:

• Usability defects are very tangible. Every time we find a bad user interface, we know how it could have been better if only the designers (or their managers) had followed well-documented usability guidelines. It's not like having to invent a faster airplane, which requires new research. Most usability improvements require only that companies implement what's already known. That's why it's extra frustrating when they don't.
• Although usability applies to anything with a user interface — from elevator controls to consumer electronics and remote controls — it's most prominent for websites and software applications, which run on computers and are thus compared with progress in computer technology. Such progress tends to be rapid:
  • Moore's Law for increasing computer power: 60% per year.
  • Nielsen's Law for growth in Internet bandwidth: 50% per year.

When we're part of a business that routinely sees 50–60% progress per year, our 6% usability improvement rate seems puny indeed.

So, why is usability less like technology and more like all other fields of human progress? Because usability is about humans, not computers. We're designing around the fixed limitations of the human mind, and we have to improve websites and products within the constraints of organizational inertia. It's not enough for a designer to attend a usability seminar and learn the guidelines for making websites easier. The designer also has to convince the marketing VP, which can take years. (It takes about 20 years from a company starts doing usability until they have the process right.)

So is the glass half full or half empty? I say half full, because we have indeed made progress in usability, and we're doing so at a faster pace than most fields. Still, some other areas of human progress are moving faster than us, so we can't rest on our laurels. Usability progress could easily speed up to, say, 7% per year if we improved our ability to communicate the power of user-centered design.

Learn More

Full-day seminars on

• Current usability guidelines for websites
• User testing
• 3-Day Camp on usability in practice (only in a few cities)

at the annual Usability Week conference.