Jakob Nielsen's Alertbox, April 25, 2005:

Formal Usability Reports vs. Quick Findings

Summary:
Formal reports are the most common way of documenting usability studies, but informal reports are faster to produce and are often a better choice.

I recently asked 258 usability practitioners which methods they use to communicate findings from their studies:

42% produce a formal written test report with full details on the methodology
36% write a "quick findings" report
24% circulate an email that lists the study's top findings
15% disseminate a spreadsheet of the findings
14% enter usability findings into a bug-tracking database
21% conduct a meeting in which they offer a formal presentation of the findings
27% conduct an informal meeting or debriefing to discuss the findings
1% show full-length videos from the test sessions
4% show highlights videos from the test
3% create and display posters or other physical exhibits

There's no one best approach to reporting usability study findings. Most people use more than one method, depending on their corporate culture and usability lifecycle approach.

That said, the survey clearly found that formal and brief reports are the two dominant means of disseminating usability findings. Both approaches have their place.

When to Use Quick Findings Reports

You can maximize user interface quality by conducting many rounds of testing as part of an iterative design process. To move rapidly and conduct the most tests within a given time frame and budget, informal reports are the best option.

Preparing a formal slide-based presentation will simply slow you down, as will using videos or statistics. Instead, simply hold a quick debriefing immediately after the test, structured around test observers' informal notes on user behavior. Follow this meeting with a short email to the entire team (the shorter the email, the greater the probability that it will be read).

Some organizations thrive on formal presentations and slide-deck circulation. In my view, this is a poor method for documenting usability findings. Bullet points don't capture the subtleties of user behavior, and it's almost impossible to interpret a slide presentation even a few months after it was created.

Extremely brief write-ups work well for studies aimed at finding an interface's main flaws to drive design iterations. Such studies are largely throwaway; once you've created the design's next version, the study findings are rarely useful. As long as you're not looking to create long-term learning materials, you won't lose much by using this highly informal reporting.

Also, an old usability issue will sometimes rear its ugly head at the project's last stage. If it does, having a short description of the problem from an old study is much more useful than relying on a bullet point that mentions the issue without offering any details of user behavior.

Video clips and posters are relatively rare, but they do have a big propaganda value and thus help evangelize usability in the organization. I encourage you to try such non-document reporting formats.

When to Use Formal Reports

The formal report remains the most common format, but I think it's overused and prefer more rapid reporting and more frequent testing. The formal report definitely has its place, however, as in cases like these:

Benchmark studies or other quantitative tests. Unless you document the measurement methods in detail, you can't judge the numbers. Also, one of the main reasons to measure a benchmark is to measure it again later and compare. To do so, you need to know everything about the original study.
Competitive studies. When you test a broad sample of alternative designs, the resulting lessons are usually so fundamental and interesting that they warrant a complete report, with many screenshots and in-depth analysis.
Field studies. Most organizations rarely conduct studies at customer locations; when they do, the findings deserve an archival report that can be used for years. Also, field studies usually produce important insights that are too complex to be explained in a quick write-up.
Consulting projects. When you hire an expensive consultant to evaluate your user experience, you should milk the resulting insights for a long time to come. The outside perspective is only valuable, however, if it remains inside once the consultant has gone. To ensure this, you need a report that's both insightful and comprehensive.

All these cases require an archival version of the findings that can stand the test of time. Deeper and more comprehensive studies don't just produce a list of fixes to a design's current iteration. These studies generate insights into users' behavior and needs that are useful for years. When new people join the team, they should read several of these conceptual reports to gain background on current design directions. Understanding the big picture with respect to usability will prevent new people from making a lot of mistakes.

The best usability reports are learning tools that help form a shared understanding across the team. It's worth investing the effort to produce a few formal reports each year. One way to free-up resources and make some reports extra good is to scale down your ambitions for most of your everyday reports, keeping them quick and informal.

Learn More

How to write a usability report is covered in depth in my 3-day camp teaching Usability in Practice at the Usability Week 2008 conference in New York, San Francisco, London, and Melbourne.

> Other Alertbox columns (complete list)
> Sign up for newsletter that will notify you of new Alertboxes