Papers and Essays
1994 Web usability study
|
useit.com
Papers and Essays
1994 Web usability study
|
Each participant was tested for 60-90 minutes. During the test, the participants visited 2-4 WWW sites from a list of sites prepared by the experimenter. For each site, the participants were first asked to give their initial impression of the home page, after which they were allowed to explore the site freely. After about ten minutes of exploratory browsing, the participants were given a directed task that asked them to find some specific information that was available on the site. The sites and the directed tasks were:
| Site | Directed Task |
|---|---|
| Hewlett-Packard | You want to see the photo of the garage in which the company was founded |
| IBM | You want to know how many hard drives you can put into a PC Server 50 |
| Microsoft | You are interested in knowing how Windows'95 will support access to the Internet |
| Sun Microsystems | You are interested in knowing more about the Spring Distributed Operating System Project (a project trying out alternatives to Unix) |
| Time-Warner | You want to read the profile of the co-creator of Mosaic. His first name is Marc and his last name starts with an A, but you can't remember the exact spelling |
The users were tested using NCSA Mosaic version 2.4 for X Windows running on a SPARC 10 workstation with a megapixel 256-color display. The workstation had an indirect connection to the Internet through a firewall with transfer rates of approximately 10 kilobytes per second. This is about five times faster than a modem and about the same as ISDN and thus represents best-case performance for home users, though on the slow side of what one would experience from an office computer that was directly connected to the Internet. When using the Sun WWW pages, the users accessed an internal mirror server to which the workstation had a direct network connection, and transfer rates were much faster for the Sun pages (around 65 kB/s) than for the pages retrieved from the Internet through the firewall. The users commented several times that access to the Sun pages was faster than their normal experience and that access to the non-Sun pages seemed a little slow. Thus, we can assume that the network speeds used in this study were approximately (though not exactly) the same as those normally experienced by the users.
Many other sites were considered and were rejected either because they were similar to the ones chosen or because we were sure that we did not want to emulate their designs. For example, the home page for HotWired used icons and labels that were essentially impossible to understand. Such a design may be appropriate if "hipness" is the essential quality to be communicated, but I felt that our design should involve qualities like ease of use and powerful computers. One possibility would be to follow the example set by eWorld and design the corporate Web page to mirror the graphical user interface of our desktop software, but we were not aware of the eWorld Web site when the comparative usability test was planned, so this option was not tested, except for the observation that users liked a similar design feature at Microsoft's site.
On the Sun home page, all users clicked on the Santa Claus icon first due to its prominent position at the top left corner of the page. Also, several users expressed a liking for the untraditional icon design with the button graphically protruding from the image. All users overlooked the row of flat buttons on top of the Santa button. Several users were actively looking for a "new at Sun" button and a search feature but did not find them. The design used for the Sun home page makes the button bar look too much like captions or headers for the top row of much more prominent icons.
All users complained when they retrieved pages with a screen or more of unstructured text. They plainly did not want to read much, and at best scanned the text for important or relevant paragraphs. The users liked information that was presented in lists that were easily scannable, especially when icons were used to indicate the different parts of the list (see the figure from Hewlett-Packard's site). Also, users liked the use of a "New" indicator to highlight new information in a list and the use of horizontal rules (<HR>) to partition a page. Long pages (more than a screen) were only deemed acceptable when users could quickly decide to ignore most of them and focus on the relevant parts (e.g., a list of Microsoft networking products organized by operating system).
In one case, a user retrieved a page that was very similar to one he had seen before and wondered whether the two pages were in fact identical. This incident indicates a potential problem with representing the same information multiple times with slight variations.
Users also sometimes scoped their search incorrectly: in one case, a user was looking for information on the Sun server and came across the Catalyst project page, which included a search form for finding information in these pages. The user happily used the Catalyst search in the belief that it searched the entire Sun server and not just the Catalyst database. The conclusion from this observation is not to offer search features that are limited to a subset of the information on a sever. If scoped search is needed anyway for some reason, it should be made very clear to the users what information is being searched and what it not being searched and there should be a direct hyperlink to the global search page. Also, users sometimes identified the need for search after they had traversed several links and were deep in the hierarchy. They expressed a desire to be able to start a search from their current location and not have to navigate back to the home page first.
On the Time Warner page, one user was surprised when the "Virtual Garden" button pointed to information for home gardeners. The user had expected it to be a list of links to other interesting Web sites to visit. Users may have become so accustomed to the heavy use of metaphors in many WWW designs (and systems like Bob and Magic Link) that they expect design elements to be metaphorical rather than literal. The lesson for user interface design is to consider not just the first-level interpretation of proposed design elements but also whether they could be misconstrued as inappropriate metaphors.
In a few cases, users retrieved information that caused a PostScript viewer to be launched without warning. The users complained that the WWW page had not warned them that the link would download a PostScript document rather than jumping to another hypertext screen and they also did not want to read the PostScript document (as mentioned above, the users disliked long texts). Even though the users disliked links to PostScript files that were displayed in a separate viewer, they did like the possibility of using FTP to retrieve additional information, executables, patches, or other non-hypertext files.
Users consistently praised screens that provided overviews of large information spaces. For example, they liked the HP navigation screens, Microsoft's home page with its extensive listing of server content, and the overview page for Windows'95 (see figure at the beginning of this report). Some users did complain that the text in the Microsoft home page overview was too small.
Several users who were getting lost coped by analyzing the file names with full directory paths given for HTML files to understand the server hierarchy and their current location. Unix experts may be capable of doing so, but general users would need a simpler mechanism for understanding the information space.
Broadly speaking, it seemed that users disliked 1x1 inch thumbnails but liked 1x2 or 2x2 inch thumbnails. To communicate graphical information in a one-square-inch space one should thus use icons and not thumbnails. The general conclusion regarding thumbnails, though, is that they should communicate some information to the user and have a clean and uncluttered appearance. Thus, the exact size of a thumbnail should be a function of the complexity of the original image. If it is impossible to represent an image clearly with a thumbnail then it will be better to use a textual description.
Users had little patience for server error messages. One user said that he might try retrieving the page once more, knowing that servers sometimes got overloaded, but if he got a second error message from the same server then he would never visit it again: he would assume that it was too unreliable and/or too poorly maintained to be worth his time.
Users were also very disappointed when they came across evidence that a server was not being kept up to date. For example, several sites listed talks or conferences that had already taken place as if they were still upcoming. In related comments, users praised the way some information on the Sun and Microsoft servers was labeled with dates to indicate how current it was. Users also liked the use of the month name to indicate the currency of the information on IBM's home page.
The users also liked the forms that were provided at several sites to allow them to give feedback to the Webmaster. This observation may also be related to the "human touch" phenomenon mentioned above since having the form makes the users feel more in contact with the actual people who maintain the server as opposed to being just consumers of a flow of information.
Users did not mind online surveys and questionnaires as long as they were at most one page long. Users frequently complained when they came across pages with questionnaire forms that did not fit on the screen. Users expressed a wish to have as much information as possible filled in for them. For example, a user who had already filled in his name and address on one questionnaire wanted the computer to automatically insert that information in another questionnaire he retrieved from the same server a few minutes later. One user wanted to be able to enter his email address to subscribe to notices about updates for a certain page that he found particularly interesting.
This usability problem cannot be solved in the design of any individual server but must await a change to URNs or some other addressing scheme that takes mirror servers into account.
Two conclusions can be drawn from the users' extensive user of parallel sessions: WWW users are very impatient and do not like waiting for information to be retrieved over the Internet. As far as possible, one should avoid having them wait (since their attention will wander).
One cannot assume a one-to-one correspondence between the stream of WWW requests received from a user and that user's hypertext browsing, since two or more browsing sessions may be interleaved.
One user had read in the newspaper that Sun had information about the Rolling Stones online and searched for that information during the study. Knowing about certain information from paper-based sources is one way that users can get prompted to look for it, and the URL should be given in those printed sources as far as possible. Some users mentioned that they would not want to spend time during the day to read extensive information but that they might print it out or email it to themselves to read later. With current technology, most printouts will be restricted to grayscale graphics and much email will be restricted to ASCII text, so the information should be designed to be useful in those formats.
I found that users wanted search and that global search mechanisms should be globally available. Even so, users were poor at specifying search strings and they often overlooked relevant hits. Thus, we cannot rely on search as the main navigation feature. Navigational structure and overviews are necessary to avoid user confusion and should be provided both in the large (server structure and location) and in the small (structure for the individual pages with iconic markers for the various types of information). Users liked the feeling of being part of a two-way communication with a site staffed by real humans and not just the recipients of a stream of bytes coming in over the net. Care should be taken to provide a "high-touch" feeling in addition to the "high-tech" image of a WWW server.
Introduction (Added 1997)
This reports summarizes results of a usability study of several Web sites I conducted in the beginning of December 1994. Users were observed as they browsed the Web sites of Hewlett-Packard, IBM, Microsoft, Sun Microsystems, and Time Warner. The report has only been very lightly edited and thus represents my thinking about Web usability in 1994. In fact, the report was originally written for distribution to the rest of the Web team on paper since we were not heavy intranet users in 1994, despite having designed SunWeb a few months before this study.The report is of some historical interest, both because it includes screen captures of several famous early websites and because it is one of the first formal usability studies of the Web. In fact, it is remarkable how well the findings and conclusions in this report hold up in the light of the current state of the Web. People often talk about how the Web changes on "Internet time", but usability issues seem to change much more slowly since they stem from human capabilities and interests.