Usability Testing of Advanced Homepage Concepts

Sidebar to Jakob Nielsen's article on Sun's 1997 Web design

Initial work on the new design was based on a parallel design exercise where six very different Web site concepts were developed and tested with representative users. The most creative of these concepts was probably the "crystal ball" home page which is shown here, but many of the other designs were also very interesting. The usability study showed that users were not interested in far-out designs. Instead, they preferred the more straightforward of the designs, including one with plain buttons and small feature story illustrations that eventually developed into the current home page.

Crystal Ball Concept

The home page for the Crystal Ball concept. This page was never implemented but was tested as a paper prototype where users were asked to use their finger as a mouse and point to the things they wanted to click on and describe what they thought would happen.

Positive terms used by users to describe this concept:: "more of a design" (than other concepts shown), creative thing, artistic, looks neat and appeals to engineers (said by an engineer), Sun is in the middle of technology, Sun is in there, Sun combines all the things shown in the sphere, very striking, "I am definitely drawn right to the shape", is using the medium properly, avant-garde, slick
Negative terms used by users to describe this concept:: A lot jammed into a small space, communicates confusion, a little bit busy in the center, "hard to get a strong feeling of what the company is doing", hard to read, confusion, busy, "a company with poor graphic design", looks pretty confusing, symmetry isn't there, very busy, "will take long to load", catalog, too much extra glitz, if graphics look bad on a low-resolution screen then it will reflect poorly on Sun, tilted menus slow down reading, pretty confusing, text is all over, not an appealing format to me, feels locked in, not intuitive, this is insane, a waste, too much extra glitz, gets in the way of the content, just a pretty picture, piece of eye candy, pure marketing piece that's just staying "buy our stuff", takes too long to figure out
Usability problems:: Most users found it difficult to read the "Shopping Forums" text that went vertically from the bottom to the top. Several users didn't see the Sun logo at first. Most users complained about not knowing what would happen if they clicked in areas where multiple lenses were overlapping. Some users had difficulty reading the overlapping words "Hardware" and "Products" on the product listing page, though one user liked this design. Some users complained that the interactions styles varied too much between pages in the same design (as one user said, "I would like to be able to reuse my learning", indicating users' awareness of Web design concepts).

Web Users Get Design-Savvy

It was striking how sophisticated the users were with respect to analyzing Web user interfaces. Without special prompting, users discussed navigation metaphors and other user interface design issues in relatively knowledgeable terms (including frequent use of the word "navigation"). In my early Web studies in 1994, Web user testing was much like any other usability studies where the users just performed the given tasks I gave them without any meta-analysis of the structure of the interface; this situation has clearly changed. Two likely reasons for this new design awareness are that users move between scores of Web user interfaces every week (and thus have ample opportunity to compare and contrast UI approaches) and that the trade press has started carrying a fairly large number of articles about Web design.

Having users understand Web design to some degree has the practical implication that they are not going to be impressed by gimmicks since they can see through them. The users several timers referred to design elements in our mock-ups as being initially interesting but not useful for long-term use. Another practical implication is that highly useful and functional design will carry brownie points with users beyond their increased ability to use the site. As one user said: "the Net being the great Equalizer, you can tell who spends how much on their home page" and how serious and competent they are with respect to the Web.

Web as Genre

It was also striking that users had developed an understanding of Web conventions and the traditional way Web pages and Web navigation work in most sites. Any deviations from these conventions in our advanced concepts received negative comments from the users. This is not to say that one can't break new ground but it is necessary to be careful not to deviate too radically or without motivation from the established ways of the Web. The Web is establishing expectations for narrative flow and user options and users like having pages fit within these expectations. A major reason for this evolving genre is that users frequently move back and forth between pages on different sites and that the entire corpus of the Web in some ways constitutes a single interwoven user experience rather than a set of separate publications that are accessed one at a time the way, for example, traditional books and newspapers are.

Scrolling Tickers

Almost all users disliked the scrolling tickers (marquees) in some of the prototypes. Users complained that they were hard to read and time-consuming to interpret. One user kept missing the beginning of the text and thus had difficulty understanding what the message was about. One user said that he tended to ignore such text with the explanation that "I have never seen any information in crawling text that had any interest to me." One more indication that users can see through gimmicks and that they have an explicit understanding of Web design and what they like and don't like on the Web.

Feature Story Usability

Users generally liked having news stories on the home page as opposed to reserving the page strictly for navigation. Some users expressed a preference for limiting the news coverage to a small number of stories in order to avoid scrolling. Several users related recent experiences where they had had problems with vanishing news stories: they had seen an interesting story on the home page but could not find it at a later time when they needed the information. Because of this finding, we decided to keep previous feature stories at the bottom of the home page for some additional time and to prominently feature the link to an archive of all old feature stories.