Test with Real Users

Nothing in this book means anything unless you test with real people. Moreover, you need to test with real users: not your colleagues, not your boss, and not your partner. You need to test with a diverse mix of people, from the widest section of society you can get access to (within your target audience, of course).

User testing is an essential step to understanding not just your product, but the users you’re testing—what their goals really are, how they want to achieve them, and where your product delivers or falls short. You’ll not only understand your users better, but you’ll reduce development time by short-circuiting the feedback loop and getting problems fixed much earlier in the product life cycle.

It’s never too early to start testing—an unfinished prototype or even paper prototype (cards or notes that you move around on a desk) can yield valuable insights—so get your product in front of users as soon as you can. Additionally, you can save yourself (and your developers) a huge amount of work and rework by heading in the right direction early, rather than making lots of false starts.

So, what are you testing? User tests are, in themselves, a broad spectrum of activities ranging from “guerilla-style” tests—where you approach a random person and ask them to perform a task in the app—through to specific feature-based tests, where an expert user (usually with domain knowledge) is asked to perform a complex task. Either way, you need to start with an idea of what you’re testing, tuned to both the complexity level of the product and the domain knowledge that a user needs to operate it.

A few of the most common usability testing methods are:

• Guerilla testing: informal and ad hoc, as mentioned above. This method is cheap, rapid, and great for getting an early steer on your proposed solutions.
• Lab tests: performed in controlled conditions with a moderator. This method provides greater insight into the users’ motivations, but they can be costly and slow to yield results.
• Remote testing: performed unmoderated, where the user is left to their own devices. This method does not provide the same depth of feedback, but you can scale these tests up to thousands of participants.

There’s a myth that user testing is expensive and time-consuming, but the reality is that even very small test groups (fewer than 10 people) can provide fascinating insights. The nature of tests with a small number of participants is that they don’t lend themselves well to quantitative analysis, but you can get a lot of qualitative feedback from working with a small sample set of fewer than 10 users.

There’s research to show that testing with as few as five users will uncover 85% of usability problems in a single test. This startlingly high number is arrived at thanks to the Poisson distribution and some math.

Why you only need to test with 5 users, NNG: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Too often, products aren’t tested, the thinking being that “we’ll just hear what users don’t like and fix it.” The problem is that your users won’t tell you; they’ll just leave. The near-infinite choice of products and services on the web, app stores, and a myriad of devices means that the user has no incentive to stay, complain, and help you to improve your product—it will simply fail.

The cost of switching to an alternative for a user is almost zero—a quick Google for a competitor and they’re gone for good. Test with real users and listen to them, and you’ll build something they love.

Learning points

• Test your product early and with real users within your target audience
• Test with a diverse group of people from different ethnicities, ages, genders, and backgrounds
• You only need to test with a small group to get huge benefits