Jakob Nielsen's Alertbox, June 11, 2007:

Change vs. Stability in Web Usability Guidelines

Summary:
A remarkable 80% of findings from the Web usability studies in the 1990s continue to hold today.

As Web usability testing enters its 14th year, it's worth asking how early results have held up to recent user research.

10 years ago, I wrote an article on the changes in Web usability from 1994 to 1997. A few of my original findings were no longer valid a mere 3 years after they were issued. But most of the 1994 guidelines held true in 1997 -- and they're still correct today.

Considering how primitive websites were in 1994, it's striking that most of these initial usability guidelines remain valid for today's sites. It's even more impressive when you consider that the Web currently has 120M sites, and my very first study tested only 5 sites with 3 users. This tiny, exploratory study's outstanding outcome and endurance is testament to the power of qualitative usability methodology.

In 1999, I published my book Designing Web Usability: The Practice of Simplicity. I based its guidelines on tests of about 100 websites with 200 users. Over the 5 years between my first test and publishing that first comprehensive book, we learned a lot about Web usability, making it possible to provide sound advice for successful Web design.

Reassessing Old Web Usability Guidelines

It's reasonable to ask whether the guidelines in Designing Web Usability continue to be valid. After all, our current Web usability guidelines are based on testing 831 websites with 2,744 users in 16 countries. The new research is vastly more thorough than the studies I did in the 1990s and has identified more than a thousand new guidelines.

My recent book, Prioritizing Web Usability contains a chapter assessing the guidelines from the 1990s in light of the new findings. (The rest of the book prioritizes the thousands of new usability guidelines according to their importance for business success.)

In the new book, I offer a section on each of the early Web usability guidelines and judge them in relation to current research findings. I award each guideline a number of skulls, depending on how important it is for today's users:

• No skulls means that the issue is no longer a problem.
• 1 skull indicates guidelines that are now minor issues.
• 2 skulls signify medium-impact usability problems.
• 3 skulls indicate issues that still cause major problems in current user testing.

Let's define our baseline as the total number of potential skulls if all the 1990s' guidelines had remained major problems and gotten 3 skulls. In fact, several guidelines got 0, 1, or 2 skulls, because various issues have become less problematic over time.

Why Guidelines Change

I remove skulls from a guideline for three reasons that align with the three ways in which a usability issue might be diminished in the modern world.

• Technology improvements: Better browsers, faster bandwidth, or other beefed-up technologies make a particular design idea easier to stomach.
• Behavioral adaptations: As people grow accustomed to certain interaction techniques, they adapt their behavior, making the techniques easier to use.
• Designers exhibit restraint: A design element might remain problematic in principle, but Web designers learn to avoid its most obnoxious forms. The element thus causes fewer problems, simply because it's being abused less often.

The following pie chart shows how the original "skulls" (i.e., usability problems) from the 1990s are distributed today:

Fate of Web usability guidelines from the 1990s.

Conclusions from the pie chart:

• More than half of the usability findings from the 1990s remain in force.
• Only 10% of the original usability issues have resolved because of improved technology. Yes, there have been many advances in Web technology, but they typically haven't addressed the real issues that cause users to get lost, misinterpret sites, or be annoyed.
• More skulls were taken away because of changing user behavior than because of technology improvements. In principle, I think it's unreasonable to expect humans to adapt to computers, but it does happen, and this is one way that awkward user interfaces eventually become easier to use. The Web is easier for people to use today than it was in 1994, simply because people have more experience with its conventions, even when they're bad.

Finally, let's consider the pie slice labeled "designers showing restraint." Of the guidelines from the 1990s, 22% are less of an issue today because designers have learned to be less abusive. But truly, that doesn't make the underlying guidelines invalid; it simply makes them less important.

Take, for example, the guideline to avoid splash screens. The best-selling Web design book in the 1990s strongly advocated splash screens, which is why I had to campaign so strongly against them in 1999. I don't bother mentioning splash screens in my current courses, however, because no decent designer would open a website with a splash screen. However, a few clients still request splash screens, so we keep the guideline on the books so that designers can point to documented usability research when warning clients against this annoying design element.

In other words, the 22% of the old usability problems that are avoided because of current designers' restraint are like unexploded bombs -- they can go off at any minute if future designers forego restraint or clueless clients insist on bad design.

Stability of Usability Guidelines

When we add the points that remain in force to those that represent designer restraint, we find that 80% of Web usability insights from the 1990s are still current or potential problems today.

I think 80% is a fairly good performance, considering that my 1999 book was based on data from only 200 users.

On the other hand, we can compare the Web usability guidelines with the durability of application design guidelines from 1986. When I analyzed these guidelines, I found that 90% continue to be valid. This higher score is all the more impressive when you consider that the application guidelines are 13 years older than my Web guidelines. There are two reasons the application guidelines had higher durability than the Web guidelines:

• The 1986 guidelines were based on more than 25 years of research into application usability. Studies are cited from as far back as 1961. In contrast, my 1999 Web guidelines were based on only 5 years of research. Still, it was worth publishing the book in 1999 instead of waiting another 20 years, because it was unthinkable to let the horrible state of Web usability continue until 2019 without fighting back.
• The Internet has changed more rapidly than computers in general. Even if most of these changes don't impact usability guidelines, some of them do. As a result, we've had to modify the early Web guidelines to a greater extent than what's usually necessary for usability guidelines.

Overall, though, whether you look at application or website guidelines, usability guidelines remain remarkably stable across decades. That's because they depend on human characteristics, which don't change that much.

Newer Web usability guidelines are likely to prove even more stable than the findings from the 1990s. So far, we have not revised a single guideline that we've discovered in research since 2000. Each time we study something again, the guidelines are reconfirmed. We continue to discover new guidelines and to retract some of the 1990s' guidelines, but all the guidelines documented since 2000 remain in force. I'm sure we'll eventually get a test result that causes one to be retracted, but this hasn't happened yet.

Learning from History

History shows that the enemies of usability never give up, even as they fight an ever-retreating battle.

One of the enemies' most insidious arguments goes as follows: "Yes, usability advocates were right in the past, but they are wrong now." By thus granting the obvious validity of well-established principles, they gain some credibility in rejecting recent user research.

• In 1995, the enemies of usability said that user testing might be a good way to improve applications, but that it didn't work for websites. (Ironically, these same people say the opposite today: that usability methods might work for websites, but not for applications.)
• In 2000, the enemies of usability said that the usability guidelines might be fine for static websites, but that I was crazy in warning against excessive use of Flash, which "obviously" made for more exciting, dynamic sites.
• Now, the enemies of usability say that while I was perhaps right about the early Flash problems, it's not reasonable to apply traditional usability guidelines to " Web 2.0" which proponents claim will revolutionize everything and do away with all that we know. (No, it won't.)

The people who now say that the usability findings from 5 or 10 years ago are obvious are the exact same people who vehemently rejected them at the time.

I safely predict that in 2012, these same enemies of usability will write blog postings saying "sure, Jakob Nielsen might have been right in 2007, but he doesn't get the fancy new stuff we do here in 2012." Those of us who bother doing the user studies will be laughing in 2017, when everybody finally agrees that the research findings from 2012 were right on target. He who laughs last, laughs best.

Learn More

I discuss more of the basics that users really want in my full-day tutorial on Fundamental Guidelines for Web Usability at the Usability Week 2008 conference in New York, San Francisco, London, and Melbourne.