Jakob Nielsen's Alertbox, June 13, 2005:

Archiving Usability Reports

Summary:
Most usability practitioners don't derive full value from their user tests because they don't systematically archive the reports. An intranet-based usability archive offers four substantial benefits.

I recently asked 245 usability practitioners what they do with their old user testing reports. Their answers were as follows:

12% keep reports in a knowledge base;
27% keep reports in a single online document collection;
29% keep reports online in various locations, so they have to track down reports when they need them; and
33% don't keep old usability reports online at all.

In other words, most organizations have a disorganized archiving policy. This is a shame, because there are many benefits to having easy access to past user research results.

Archive Strategies

I don't recommend establishing a formal knowledge base of usability reports. The current state of knowledge management (KM) is primitive, and you're unlikely to see benefits that are worth the cost and hassle. The exception here is if your company has already invested in KM for other documents, and you can piggyback on the existing system.

At a minimum, you should establish a centralized intranet usability site and populate it with a permanent repository of all your usability reports. When people have to search the intranet for usability reports in individual project subsites, they'll often fail -- or they won't even know what to look for because there's no single place that lists all available usability results. Even worse, if past project owners are the only source of results, you risk losing valuable institutional memory when these individuals leave the company or get reassigned.

Whenever you go to the trouble of creating an in-depth formal report with detailed analysis, you need to maximize your return on investment by encouraging future use of the insights. You should also archive informal usability reports. These "quick findings" and emailed summaries are important during usability projects, and while not as useful in the future as more detailed reports, they will have value then as well.

Archive Benefits

Having good archives of past usability findings offers four main benefits, ranging from the tactical to the strategic:

When new people join the project, past findings give them an easy way to review what's already known about users. This gets them up to speed quickly and will prevent them from repeating mistakes that you've already documented in past testing.
Projects can often stretch over many years, especially when there are multiple releases and modifications. As the years pass, different people get involved, especially if you outsource designs to an ever-changing set of agencies. In this situation, usability reports often provide the only design rationale for why certain approaches were chosen and others abandoned because they didn't work for users. If you don't know what was tested in the past, you might repeat old mistakes. If you don't know what has been learned about user needs, you'll be less capable of supporting those needs in the next version. Also, implementing usability insights can take a long time: an old consulting client recently called us up and said, "We're almost done implementing the recommendations you gave us two years ago." If they hadn't saved the old report and referred to it during years of design work, they wouldn't have gotten their full money's worth.
Individual findings can be generalized to usability guidelines once you observe them repeatedly. One of the most powerful ways to increase your usability group's productivity is to develop customized guidelines for your specific type of user interface.
After accumulating a large number of reports, you can track trends over time to assess your usability work's long-term impact: are you getting better or worse? You can also perform meta-analysis on cross-project data to gain insights that transcend individual projects. In my presentation on user satisfaction, for example, I use data from our tests of 209 websites to show the distribution of satisfaction ratings and calculate the score a site needs to have above-average customer satisfaction relative to the rest of the Web. Similarly, within a company, you can aggregate scores across multiple studies to discover the success rates, task times, or satisfaction scores a new project needs to be considered acceptable.

On a personal level, archival usability reports are a great learning resource for improving your usability skills. Although the best way to become a great usability professional is to conduct numerous user tests, reading the findings from other studies is a close runner-up. Thus, when you keep your reports to yourself or otherwise make them difficult to access, you shortchange your colleagues and your organization as a whole.

Learn More

Usability reports are covered in depth in my 3-day camp teaching Usability in Practice at the Usability Week 2008 conference in New York, San Francisco, London, and Melbourne.

> Other Alertbox columns (complete list)
> Sign up for newsletter that will notify you of new Alertboxes