Usability Testing of WWW Designs

By Jakob Nielsen, May 1995
Sidebar to article about the design of Sun's website in 1995.

Photo of usability lab set up to run three different types of studies
Usability lab setup: click on a test area for more detail.

We used several rounds of usability testing to improve the user interface for Sun's new WWW pages. The initial tests focused on the home page and were done using paper prototyping. We also conducted competitive usability tests where users were observed while browsing WWW pages from other companies.

The category grouping used for the main icons on the home page was derived based on a card sorting study: several users were given a series of index cards with various concepts from the server written on them. The users were then asked to sort cards into piles, placing cards that seemed similar to them into the same pile. Concepts that were placed in the same pile by many users were deemed sufficiently similar that we should place them in the same category in our new design.

Of course, we also used traditional user testing, where users were asked to use a running prototype of the new pages. Sometimes, rather primitive HTML mock-ups were used, with many dangling links or links that pointed to pages on the old server that used an obsolete user interface. Even so, we were able to learn a lot about how people use WWW pages. Usability testing was conducted both in our usability lab in Mountain View, CA (see the photo on top of this page) and in Sun offices in several other countries around the world (to assess international usability).

In addition to studying our various new design ideas, we also conducted a usability study of Sun's old WWW design. Of course, we were well aware that it contained many usability problems (e.g., inconsistent headerbars and several very strange imagemaps) but we still wanted to learn what worked well and what worked less well in the old design. One interesting finding was that the top button-bar did not look enough as buttons: the design did not have a clickability affordance but was seen as mere decoration by most users. We redesigned this specific aspect of the old design immediately without waiting for the full redesign to come online. The following figure shows before and after versions of the top row of buttons in the old design:
Two rows of buttons with a top button-bar

Changing the button design as illustrated above resulted in 416% increased use over a two-month period (January-March 1995). Considering that the use of the server in general increased by "only" 48% in the same period, there can be no doubt that usability engineering worked and resulted in a significantly improved button design.

Reference

For more info on how to use these methods (as well as many more usability engineering methods), please see my book Usability Engineering and my report on Return on Investment from Usability.