Usability testing methodologies

Usability tests can be run in different ways and for various types of studies. Let's first look into the different ways (or methodologies) of executing a usability test. These are mainly differentiated by the following characteristics:

• In-person versus remote 
• Moderated versus unmoderated

In-person or lab usability testing

This is the traditional way to run a usability test: users who meet the user profile requirements are invited to participate in a usability study at a lab. This methodology is almost always moderated, meaning that there is always a UX researcher present who moderates the user sessions. 

In-person usability testing

We are listing this methodology for completeness, but this book only covers remote methodologies.

Remote usability testing

While in-person usability testing has been around for some time, remote usability testing has only more recently become feasible due to advances in technology and the availability of new tools. 

Connectivity and the internet have made it possible for us to run usability studies remotely using software that allows the moderator to view and capture the participant’s screen, hear their audio input, and view their faces. Enhanced distribution and recording tools have also made it possible to target participants offline and receive high-fidelity user videos and feedback.

Remote usability testing

Remote usability testing allows companies to gain insights into user behavior in their natural environment and on their own schedule. It involves fewer logistics, allows participation regardless of location, and is quicker and cheaper to execute compared to in-person studies, while still delivering valuable insights and feedback. In today's globally connected world, quickly getting local insights is imperative for providing the same standard of user experience while remaining culturally specific.

Advantages of remote methods

We are raving fans of remote usability testing (you might have guessed this by the fact that we wrote a book about it) for the following reasons.

Extended reach

Remote studies make it easy to include participants from different geographic locations in the same study. Essentially, anyone with a smart device and an internet connection can participate. Imagine that the product under testing is a social app aimed at allowing people to connect with each other when in proximity of another person using the same app. The typical users are city dwellers in densely populated capitals around the globe. Setting up in-person studies in each of the major cities would be logistically complex and expensive. This is where remote studies provide the biggest benefit. 

Typical devices

In a remote study, study participants use their own devices. That variety of devices combined with operating system versions (and, optionally, browser versions) and connectivity speeds is near impossible to match in a lab setting. Imagine a wayfinding app that provides directions to a selected target which is meant to be rolled out globally. A remote, globally distributed study will allow the study sponsor to gather realistic feedback with regards to how the app deals with the connectivity available locally on typical devices.

No travel required

It is easier to recruit study participants when there is no travel required as it lowers the hurdle for participants to take part in the study. A remote usability study can be completed within as little as half an hour. Imagine a study for a restaurant supply ordering site where the target user group is restaurant chefs. These folks work so many hours that it is very hard to convince them to travel to a lab. Or imagine a company that develops products that are mostly sold in rural areas, such as farming or harvesting equipment. It will be challenging enough to find participants for this study, let alone asking these participants to travel to an in-person study location.

No travel costs

Requiring participants to travel to a lab not only substantially increases the time required for the participants, but also their compensation. As remote studies eliminate the need to travel to a lab location for both the UX researcher and the participants, the costs associated with travel do not apply.

No lab costs

As no lab space is required, it follows that there are no costs for it. In a remote study, the study participants are either in an environment that they select or one mandated by the study (for example, if the study requires that the participants complete the study at their place of work), but they are never required to come into a lab.

No lab environment

A remote study allows the participant to remain in their own environment using their own devices. Any UX study is by definition an artificial event since participants knowingly sign up to provide feedback. We can reduce the level of artificiality around a study in a number of ways. One significant factor is the participant's surroundings and the devices they are using. The participant will feel more comfortable in their own environment than in a lab environment. Imagine that the website being tested is a banking website and the target users are existing customers. If the study requires that real tasks are completed on the website, participants will probably feel more comfortable displaying their banking details on their screens while in their homes than in a lab.

Familiar equipment

In a remote study, the study participants are required to use their own devices to complete the tasks. This is another significant advantage of the remote methodology because it removes the hurdle of participants having to familiarize themselves with new hardware and/or software. Imagine the usability test of some gaming software. Gamers can be very particular about the equipment they use to play. The remote study allows them to play using the devices they are familiar with, and thus eliminates the hurdle of first having to get used to devices provided to them in a lab. Another relevant scenario is users who require assistive technology, such as a screen reader; using their own devices will ensure that they do not first have to configure settings to suit their individual needs.

User's natural environment

Capturing the participant's own environment can also help discover important aspects of how users might interact with the product. Imagine testing a TV-streaming device. The study sponsor will be interested in understanding the individual setup of the participants' environment, such as where they sit when using the device, what the distance to the TV is, or what lighting is available. These settings are difficult to replicate to everyone's individual taste in a lab.

Interruptions can also provide valuable insight into a product's usability with respect to how easy it is to pick back up after an interruption. This is relevant for any interface that is frequently or typically used in an environment or at a time when interruptions might occur, such as a colleague coming over to the user's desk or the doorbell ringing, and so on.

In the wild testing

In a remote study, it is possible to send participants out into “the wild” and ask them to test an interface in a real situation. Imagine an app for browsing a conference program and navigating within a conference center. Testing the app in the participant’s home will probably not yield the same insights as compared to when they are actually using it in the conference halls and trying to find the lecture they are interested in via the built-in GPS function. Another example is a scenario where a customer is ordering ahead and picking up their order in a store or restaurant. Study participants are first asked to use an app or a website to order something and are then asked to go and pick it up. These types of studies provide insight into the broader customer experience, such as how easy it was to match up the order with the orderer, whether the order was ready at the time promised, and so on.

Larger number of participants

As a consequence of the lower costs and effort associated with running a remote study, it is possible to recruit more participants. With a larger number of participants, it is easier to determine the magnitude of any issue that is found.

Disadvantages of remote methods

We see only one main drawback to remote usability studies (although we might be biased as we are raving fans, after all).

Distributing the product under testing

When the product to be tested is not a production version, such as an app that can be downloaded from an app store, a website that is publicly accessible, or a product that the user already owns, the user will have to be adequately equipped. The study sponsor will either have to provide early versions of their designs, their software, or their product to each participant for testing.

This can usually be managed when planning the study, and we will talk about this in more detail in the following chapter. It only proves to be insurmountable when the study sponsor cannot or does not want to share these early versions outside of their lab or company for security or logistical reasons. In this case, only in-person testing is possible.

Other disadvantages

Some people will say that not being able to run the study in a controlled environment, such as a lab, is a disadvantage of remote methodologies, but we would argue that it provides a more natural context to the study. How likely is it that the user of an app will always have a completely quiet environment, with perfect lighting, free of any interruptions when they are using it? No one lives in a vacuum: the phone will ring, someone will be at the door, the participant’s child will want attention, and so on. If you remember the definition of usability testing, it states that the interface is tested by “representative users”. You can only really judge whether an interface is easy to use when it is being used by the people whom the products target in order to achieve a goal the users would actually have and in the environment that the users would naturally be using the interface in.

Another disadvantage of the remote methodologies that we try to turn into a benefit is that users might run into issues when installing or setting up the interface to be tested. When this setup is identical to how the actual product would be set up, this is valuable feedback regarding how easy it is to get started with the product under testing. However, when the study is based on early prototypes, their installation might require additional time and effort in order to get the participant set up correctly, and this is admittedly more difficult to do remotely than it would be if both parties were present in the same lab together.

Remote moderated usability testing

In a remote moderated usability study, the UX researcher and the study participant are both present and connected, but not physically in the same room. The UX researcher serves as a moderator and observer during the session. Moderated studies use the internet and collaborative software so the UX researcher can moderate and observe the participant's view of the product being tested and their facial expressions if using a webcam.

Advantages of remote moderated studies

All of the advantages of the remote methodologies listed previously apply to the moderated variation as well. In addition, the following are advantages of the remote moderated study when compared to the unmoderated variation.

Body language

If the remote moderated study uses software that allows the UX researcher to see the participant while they are completing the study, the facial expressions and potentially the body language of the participant may provide additional insight and context. This is often considered an advantage of the remote moderated study over the unmoderated variation.

Tailored follow-up questions

The ability to observe the participants in real time as they complete tasks allows the UX researcher, and any study sponsors who are observing remotely, to ask follow-up questions based on what they see. This can provide valuable, additional insights.

Disadvantages of remote moderated studies

The UX researcher has to establish a rapport without personal contact and pastries, although we would argue that the tone of the interaction can also help establish a rapport.

Remote unmoderated usability testing

In a remote unmoderated usability study, the UX researcher sets up the UX study and analyzes the results, but contrary to the moderated version, the study participants participate when it is convenient for them to do so, and without the moderation of the UX researcher. This allows the study participants to complete the study tasks where and when it is most natural to them.

Advantages of remote unmoderated studies

All of the advantages of the remote methodologies listed previously apply to the unmoderated variation as well. In addition, the following are advantages of remote unmoderated studies when compared to the moderated variation.

Natural behavior

Not being in a lab will already allow the participants to feel less inhibited and under observation, but remote unmoderated studies allow the participants to act completely natural. Imagine a music-streaming app. Ideally, the participants should use the app where and when they would naturally use it. This may be while working out or riding a bike, or even doing household chores. Participants might feel inhibited to work out or dance while being observed by a UX researcher. An unmoderated study has both the benefit that the participants can complete the study at their own convenience using the product as naturally as possible while also enabling the UX researcher to gather valuable feedback regarding the use of the app under real conditions. This is the biggest benefit of the remote unmoderated usability study.

Time zone independent

Remote unmoderated studies are independent of time zones. This is especially relevant for globally distributed studies where the study participants are located in varying time zones. In an unmoderated study, there is no interaction required between the UX researcher and the participants, thereby allowing them to complete the study at a time that suits them.

Less effort for the UX researcher

The effort for the UX researcher conducting a remote unmoderated study is significantly less than it would be if they were conducting a remote moderated study. This is because, while the effort for preparing and reporting on the study are comparable, no dedicated sessions between the UX researcher and each participant are required.

Eliminates influence

The absence of a joint session eliminates any inadvertent influence on the study participant by the UX researcher. The validity of the data could be compromised by as subtle an influence as a sigh, or obviously taking notes, for example. Moderating a usability study requires a skilled moderator who is experienced enough to know when to jump in and when to hold back. Remote unmoderated studies can be run by UX researchers who do not necessarily have this experience.

No schedules to manage

In an unmoderated study, no effort is required for scheduling the sessions between the moderator and the participants. In order to handle no-shows, moderated studies very often use "floaters". These floaters are on standby in case someone arrives late or doesn’t show up. Managing these schedules and the floaters generates a lot of overhead, not to mention the cost of having additional participants on standby. This effort and cost are absent in remote unmoderated studies.

Testing with minors

Testing with children is easier in remote unmoderated studies because parents can observe their kids and respond to their questions. Imagine a usability study where the product under testing is an educational app targeted at three- to six-year-olds. In an unmoderated study, the parents can be more flexible in interrupting the study and attending to their children if they might have lost patience or interest in the app, and resume the study at a later point in time.

Faster turnaround

The reduced effort required for remote unmoderated studies makes them faster to complete and easier to integrate into iterative testing processes. 

Disadvantages of remote unmoderated studies

There is one main disadvantage of remote unmoderated studies when compared with moderated ones.

Guidance

Participants in an unmoderated study require guidance in the form of a script to let them know what they are required to do. This script requires a lot more attention to get it right than a discussion guide for a moderated session. This is obvious when you look at the advantages of this method: The fact that the participants are geographically dispersed and complete the tasks on their own schedule makes it impossible for the UX researcher to be available to guide them through any bit of the study that may be unclear or a dead end that they may inadvertently encounter. In our experience, this is the biggest stumbling point for UX researchers who are comfortable with in-person usability testing and wish to try out the remote unmoderated variety.

The scripts need to cater to any and every outcome, given that there is no possibility to help the participants during the study. You cannot always be sure what the participants are seeing and whether they are exactly where you expect them to be, so you have to anticipate each possible path and cater for it. This is not an insurmountable disadvantage; it just requires more effort and skill on the part of the UX researcher. 

Other disadvantages

Another disadvantage of remote unmoderated studies as compared to moderated testing is the lack of immediate observation. Body posture and other physical expressions that could provide the moderator with additional insight during a study are not available. We don't view this as an absolute argument against remote unmoderated testing because you could also argue that, especially in globally distributed tests, the UX researcher may not be familiar with local gestures, and thus potentially interpret them wrongly. In India, for example, shaking your head does not mean "no"—it actually means "ok" or "I understand". Thus, asking participants to explicitly express their opinions in writing in an unmoderated study is less subject to bias than interpreting body language.

The lack of personal contact could also be viewed as a disadvantage because the unmoderated study does not allow for building a strong personal rapport between the UX researcher and the study participants. To some extent, it is possible to build a rapport in a remote unmoderated study just through how the participants are addressed in the script. As we saw with the previously mentioned disadvantage, though, what might help to build rapport with members of one culture might not work with others, or might even have the opposite effect.

One last disadvantage of the remote unmoderated methodology is that it is not possible to get reliable time-on-task values. Some unmoderated testing platforms may record the time taken on a task, but there is no guarantee that this will be valid because it is not possible to determine whether the participant continuously worked on the task or was distracted by a phone call during the execution of the task, for example.