Jakob Nielsen's Alertbox, November 5, 2007:
High-Cost Usability Sometimes Makes Sense
Summary:
Computing the net present value (NPV) lets you estimate the most profitable level of usability investment. For big projects, expensive usability can pay off.
I typically advocate "discount usability engineering" — that is, cheap and fast methods to immediately improve your user interface. But in some cases, it makes sense to invest more to get more.
When assessing your usability options, you'll often find two alternatives with drastically different costs, such as:
- Testing solely with domestic users vs. including users from one or more foreign countries to assess international usability issues
- Lab-based user testing vs. field studies at customer locations
- Testing only your own design vs. including 2-3 competitors' designs ($28K vs. $45K on our price list)
- Paper prototyping vs. a more fully implemented prototype
- Hiring usability professionals at different skill levels (and thus different salaries: $55K vs. $80K for entry-level vs. somewhat experienced staff, respectively)
- The project's expected value
- The cost of the usability alternatives
- The estimated difference in outcome for the alternatives
- The rate at which you discount future money streams to account for uncertainty
Example Calculation
As an example, we'll plug some sample values into the spreadsheet. First, let's say our project has an expected value of $1M. It might be an e-commerce site with expected sales of $5M per year and a 20% profit margin, or an intranet redesign that's expected to increase the productivity of 10,000 employees by $100 per employee per year.Note: I'm only considering the value realized during the first year after launch. This might be unfair, since projects often have a longer lifetime. For example, the average time between intranet redesigns is 3 years. If you expect your project to live for more than a year, you should obviously add the subsequent years' value (after reducing those values by your chosen discount rate to account for the added uncertainty of future time periods). Here, for simplicity's sake, I'll only look at the first year.
Next, let's estimate the difference in cost and outcome for the two usability alternatives. Let's say that the cheap approach costs $10K and that the expensive approach costs $40K.
Conservatively, we'll then assume that we expect the expensive approach to generate only 20% better results than the cheap approach. Because usability tends to double a design project's value, we'll estimate that the cheap approach will increase use by 90% and the expensive approach will increase use by 110% to bracket the expected outcome.
A 20% difference in outcome is typical when you go from domestic to international user testing, for example. Even though you learn more by testing in more countries, most findings are the same everywhere. So, if a 20% gain is going to cost you 300% more, you might think that it's a no-brainer to go with the cheap approach — but you should read on.
Finally, let's pick our discount rate: 100%. That is, whatever value we gain during the first year after project launching is only worth half as much in today's money: a $2 gain next year is worth $1 today. This huge discount rate makes sense because design projects are hugely uncertain:
- First, the project might be cancelled or fail for reasons beyond our control. Even the best design has zero value if it never ships.
- Second, all of the numbers we plug into the spreadsheet are estimates. We don't know for a fact how successful our project will be, and we don't know what user research will yield until after we've spent the money. True, usability consultants might give you firm quotes for the cost of the studies, but if you're doing it yourself or paying by the billable hour, you don't even know for sure how much the two alternatives will cost.
- No usability is the easiest to compute, since we don't have to consider the cost of the usability alternatives. We assumed a value of $1M and a discount rate of 100%, so we can easily compute the project's present value: $500K before deducting project costs. So, if design and development (other than usability) cost, say, $200K, the project has an NPV (net present value) of $300K, and should go ahead.
- Cheap usability adds 90% to the project's value, resulting in $1.9M, or $950K after applying the discount rate. We then deduct the $10K usability cost, resulting in a present value of $940K. In other words, usability has added $440K to our project's expected present value. After deducting the $200K development costs, the project now has an overall NPV of $740K.
- Expensive usability adds 110% to the value, resulting in $2.1M, or $1,050,000 after applying the discount rate. We now have to deduct $40K, resulting in a present value of $1,010,000. Thus, usability has added $510K to the expected present value. After deducting the development costs, overall NPV is $810K.
(These calculations assume that development costs are the same with and without usability. In other words, I assume we chose appropriate usability methods and set the design direction early in the project, and thus didn't have to rework any already-implemented features. If you conduct user testing late in a project, this assumption doesn't hold, and you'll have to add some cost for fixing the flaws that testing uncovered. Let's say that the cost of fixes is $50K for cheap usability and $60K for expensive usability because there's 20% more to fix. Deducting these costs doesn't change the conclusion; it simply reduces expensive usability's advantage by $10K.)
Heuristics for Choosing: Cheap vs. Expensive
If you'd rather avoid the fancy calculations, following are five general rules for choosing between cheap and expensive usability.- The higher the project's expected value, the more you should invest in usability. Usability payback comes from multiplying the project's value due to increased use (or more efficient use, in the case of an intranet). If your project's value is low, you won't earn much by doubling it.
- For example, if your project has an expected value of $5K, you shouldn't even do cheap usability, because gaining an additional $5K isn't enough to justify spending $10K. (However, there are still some ultra-cheap usability methods you can use for tiny projects.)
- The higher your project uncertainty, the higher your discount rate. In such cases, cheap usability options are typically the way to go. Yes, the more expensive usability option would offer higher gains if you were ultimately successful, but your probability of a total loss would be too great to justify the investment.
- The earlier in your project you employ usability, the more you should invest because you can implement the added findings without much rework. Late studies should be cheap; you won't be able to make many fundamental changes anyway.
- Note the tension between project stage and uncertainty: early in the project, your uncertainty is higher (which counts against big expenses), but your potential for implementing the gains is bigger (which counts for thorough studies). This is why you should save some of your usability budget for later studies, when you have more polished designs to test.
- The bigger the expected difference in outcomes, the more you should go with expensive methods. When the expected outcome difference is small, save your money and go with the cheap methods. Quantitative studies are a great example: they're very expensive and usually only worth the cost for huge projects because they otherwise add relatively little value. A similar analysis can be made for international studies: if you have few overseas customers, the added cost of testing them won't be worth the expense.
- Finally, obviously, the more expensive the high-cost methods are, the less you should use them. In our example, the NPVs will be identical if the expensive method costs $110K. Thus, if the expensive method is 20% better than the cheap method, you should use it as long as it's no more than 10 times as expensive as the cheap method. If it's 11 times as expensive, you won't get enough added benefits to justify the added cost.