Last year I found myself in a rather unenviable situation: with only one week left to run usability tests for an online poetry magazine, I was experiencing incredible difficulty locating test participants who would be willing to spend as little as 30 minutes with me. The holidays were fast approaching. And although I wasn’t a fan of remote testing at the time, it became obvious I had to bite the bullet.
Little did I know what a compelling option it would turn out to be.
In-person usability testing is the most frequently used method of product research today, hailed as “essential” since as early as 1993. Back then Jakob Neilsen explained that “testing with real users is the most fundamental usability method …it is in some sense irreplaceable, since it provides direct information about how people use computers and what their exact problems are with the interface being tested.”
Yet it’s precisely due to in-person usability testing’s prevalence that many people overlook the possibility of conducting their research remotely, in what are arguably more realistic usage contexts. In my situation, testing an online poetry journal with readers located in Australia, the US and the UK, it proved essential to include remote research. Not only was it impossible for me to travel to each location and recruit participants, it was also important to learn about — and see — each user’s behavior in their natural environment. For researchers learning about location-specific use, many will find, as I did, that remote research can prove more insightful (and therefore more effective) than its in-person counterpart.
What even is…?
Remote research is any research in which participants and researchers do not interact in-person, face-to-face. It is typically conducted via a computer or a phone, allowing the researcher to see or hear (with the aid of screen-sharing software, for example) exactly that which the remote research participant sees or hears. This makes remote research a powerful tool for studying user behavior patterns “in the wild.”
In order to gain contextual information, researchers often approach interview participants while they’re already performing a task on the actual website being tested, via a live chat feature or similar tools. This subset of remote research is known as time-aware research. It’s especially important with regards to mobile devices, where context factors heavily into a user’s experience.
During the last five years there has been a significant increase in the tools by which remote research might be conducted. Nonetheless, remote research comprises only 5% of all user research. While A/B testing and surveys are both generally accepted as types of research that can and should be conducted remotely, usability testing is rarely seen as such.
In my case, gathering real-time feedback on the usability of an online poetry magazine, remote participants were more focused on completing the task at hand than in-person participants, in part because they were not distracted by my presence. They were also more honest in their feedback, offering both negative and positive comments without any prompts on my part. Face-to-face participants, by way of contrast, were consistently verbally positive about the look and feel of the website, despite the fact that I watched them struggle to accomplish the tasks I had assigned them.
When to incorporate remote research
Despite its inherent advantages, remote usability testing isn’t right for every situation; there’s a time and a place for everything. In order to determine the best research methodology for a particular study, designers should evaluate their project against three key criteria: their research objective, their audience, and their budget.
Research objective
The first, and most important factor in deciding how to approach a study is determining the research objective. A researcher’s methodology should always depend on her objective, not the other way around. For example, if the research objective is to “obtain contextual information about the task at hand and its likely uses,” she will likely want to observe users in their natural environment. Remote usability testing, remote-access surveys, or remote A/B testing will all allow the researcher to do variations on this kind of fieldwork.
Audience
The second factor to keep in mind when choosing a research methodology is the study’s target audience. Testing remotely with students and young professionals works particularly well, since this target group is generally comfortable with online meetings and screen sharing. This is important, because in some situations the participant has to be an active participant in accepting screen recording, turning their microphone on or even circumventing firewall restrictions.
If the task at hand involves senior citizens however, who tend to have lower computer literacy, it can become complicated to set up remote testing—and technical glitches always occur. Or, if the target audience (regardless of age) uses vastly different terminology for their computer programs and websites, remote communication can become unnecessarily complicated. If, however, the audience is using a program similar to one they use daily, or if the program is intended to be particularly intuitive for first-time users, then remote testing can provide a far more accurate look at how users will approach the application.
Essentially, considering our audience requires that we answer two questions:
- Will technology overly complicate the process?
- How can we best replicate our audience’s real-life use circumstances?
Budget
Although budget shouldn’t lead our decision making process, it is a necessary consideration. Remote research is typically more cost-effective, due to the fact that it eschews travel costs in favor of software costs (e.g. screen sharing). This can be helpful in many situations, especially when conducting multiple rounds of tests.
My assignment, to look at the usability of an online poetry journal, had a very restricted budget, which made travel impossible. The objective of my research was to collect enough user research to later inform the re-design of the website, including removing or changing a variety of elements. With this objective in mind, it was essential for me to see users in their natural environments, while keeping the budget low.
Getting started
There are many different types of research available within the umbrella of remote research; the Webnographer blog recently published a list of 23! However, these remote research options are traditionally divided into two categories: moderated and unmoderated methods. Or, as Nate Bolt would say, “methods for when you do or don’t like interacting with people.”
Moderated research
During a moderated, remote user test, there is still live contact with the participant, even though it is not face to face. The facilitator can call the participant and/or interact via screen-sharing and a microphone. This way it is possible to follow the participants’ online navigation and ask follow-up questions during the test. Some tools will also connect the webcam and record a video of the participant’s face, allowing the facilitator to take facial expressions into account while interacting.
The clear advantage of using moderated but remote testing over moderated face-to-face testing is that the researcher gets to see and talk to the participant in their natural environment, which makes the test subjects feel more relaxed and potentially more honest about the experience. When comparing the results from the poetry website usability tests I noticed that my face-to-face test subjects were more polite (and less honest!) about certain features than the subjects that tested remotely.
Other types of moderated testing include online ethnography, telephone surveys, and remote eye tracking—to name but a few.
Unmoderated research
Unmoderated tests occur without a facilitator. These are also referred to as “automated” tests. For this research method to be successful, the test needs to be pre-programmed and the facilitator needs to set the user-tasks in advance. Most often the participants are asked to follow a think-out-loud protocol so they will explain their choices when performing the tasks at hand. This method is useful when there are clearly described tasks the users need to perform, with minimal room for ambiguity. The time it takes for participants to complete the task and the path they choose are at the participants’ discretion, and the outcomes can then be analysed and compared.
For example, if the research objective is to “identify how long the average user takes to follow the flow from homepage to purchase,” an unmoderated Set Task is the best way to gather results. In this situation, users can visit the site at their leisure, and will feel no additional pressure while finding their way through the site. If this were a moderated test, the user might feel an obligation to finish quickly, or would likely begin to ask questions to ensure he or she was using the site “correctly.”
Other unmoderated research types include: surveys, critical incident reports, and (the ever popular) A/B tests.
Another tool in the toolbox
After a thorough analysis of the website statistics and user data, I chose to conduct my usability tests first as moderated “live” user tests in Australia. During these in-person tests I was able to interact with the users and ask follow-up questions based on their actions. The results were complemented with four unmoderated remote tests with users in the US and the UK. We decided to move forward with the unmoderated tests in the second batch in large part due to the time-zone difference, but also in order to compare the differing results between a moderated and unmoderated test.
With the tasks clearly identified, our remote participants were able to participate in the same test I had initially attempted face-to-face, but their feedback and behavior was significantly different. They were less patient with the navigation and search options during more complicated tasks — natural behavior, which the other participants had held in due to consideration of our presence in the in-person usability tests. The combination of face-to-face and remote research gained me a new and interesting insight into user behavior for my client.
With an audience scattered across the globe, a limited budget and an exploratory research objective, our combination of in-person and remote user testing provided rich data that helped the online poetry journal to begin an informed re-design. And I came away understanding what all researchers should: research methodology must support the research objectives, and remote research is one more effective way to do so.