Archiving Usability Reports
Summary: Most usability practitioners don't derive full value from their user tests because they don't systematically archive the reports. An intranet-based usability archive offers four substantial benefits.
I recently asked 245 usability practitioners what they do with their old user testing reports. Their answers were as follows:
12% keep reports in a
27% keep reports
in a single online document collection
29% keep reports
online in various locations
, so they have to track down reports when they need them; and
don't keep old usability reports online
In other words, most organizations have a
disorganized archiving policy
. This is a shame, because there are many benefits to having easy access to past user research results.
I don't recommend establishing a formal knowledge base of usability reports. The current state of knowledge management (KM) is primitive, and you're unlikely to see benefits that are worth the cost and hassle. The exception here is if your company has already invested in KM for other documents, and you can piggyback on the existing system.
At a minimum, you should
establish a centralized intranet usability site
and populate it with a permanent repository of all your usability reports. When people have to search the intranet for usability reports in individual project subsites, they'll often fail -- or they won't even know what to look for because there's no single place that lists all available usability results. Even worse, if past project owners are the only source of results, you risk losing valuable institutional memory when these individuals leave the company or get reassigned.
Whenever you go to the trouble of creating an in-depth formal report with detailed analysis, you need to maximize your return on investment by encouraging future use of the insights. You should also archive
informal usability reports
. These "quick findings" and emailed summaries are important during usability projects, and while not as useful in the future as more detailed reports, they will have value then as well.
Having good archives of past usability findings offers four main benefits, ranging from the tactical to the strategic:
new people join the project
, past findings give them an easy way to review what's already known about users. This gets them up to speed quickly and will prevent them from repeating mistakes that you've already documented in past testing.
Projects can often stretch over many years, especially when there are multiple releases and modifications. As the years pass, different people get involved, especially if you outsource designs to an ever-changing set of agencies. In this situation, usability reports often provide the only
for why certain approaches were chosen and others abandoned because they didn't work for users. If you don't know what was tested in the past, you might repeat old mistakes. If you don't know what has been learned about user needs, you'll be less capable of supporting those needs in the next version. Also, implementing usability insights can take a long time: an old consulting client recently called us up and said,
"We're almost done implementing the recommendations you gave us two years ago."
If they hadn't saved the old report and referred to it during years of design work, they wouldn't have gotten their full money's worth.
Individual findings can be
generalized to usability guidelines
once you observe them repeatedly. One of the most powerful ways to increase your usability group's productivity is to develop customized guidelines for your specific type of user interface.
After accumulating a large number of reports, you can
track trends over time
to assess your usability work's long-term impact: are you getting better or worse? You can also perform
meta-analysis on cross-project data
to gain insights that transcend individual projects. In my presentation on user satisfaction, for example, I use data from our tests of 209 websites to show the distribution of satisfaction ratings and calculate the score a site needs to have above-average customer satisfaction relative to the rest of the Web. Similarly, within a company, you can aggregate scores across multiple studies to discover the success rates, task times, or satisfaction scores a new project needs to be considered acceptable.
On a personal level, archival usability reports are a great
for improving your usability skills. Although the best way to
become a great usability professional
is to conduct numerous user tests, reading the findings from other studies is a close runner-up. Thus, when you keep your reports to yourself or otherwise make them difficult to access, you shortchange your colleagues and your organization as a whole.