If you’ve ever run a research or usability test, you’ll know they can be tricky to facilitate. After all, you’re dealing with people; and people come with a whole host of existing preconceptions, personalities, emotions, and experiences. One thing that can help you to gain more honest and thereby useful feedback from research participants is, in fact, to lie to them.
Data is a sorted sort. Not only must it be properly contextualized and analyzed in order to bear useful information, it must also be collected and collated in a prudent fashion to begin with. Researchers go through this high level of detail to ensure the validity of their results. Dr. Marion Joppe of Ryerson University provides a more exacting definition:
Validity determines whether the research truly measures that which it was intended to measure or how truthful the research results are.
User researchers can increase the validity of their results in a variety of ways. Sometimes they conduct research “on-the-road” – known as
ethnographic research – to interact with participants in their context of use. Other researchers go as far as recreating the environmental setting in which the product will be used. For example, if testing a television or video game, they might rearrange their lab to feel like a living room (comfy sofa, pictures on the wall, etc). If the product being tested is something that’s mostly used in the evenings, they might change the lighting in the room. If participants would often be interrupted while doing a particular task, the researchers might frequently interrupt their participants during the test. You get the idea.
In his 1994 paper Usability: Practical Methods for Testing and Improvement, Miles Macleod posited the following questions to aid the validity of research:
- Are you looking at the right things to be representative of real-world use?
- Are you collecting the right data and the right amounts of it?
- Are you analyzing the raw data reliably?
The general consensus across these approaches – and Macleod’s questions – is that to increase the validity of a test requires scrutiny and planning. Though we’re all aware that planning is a good thing, what if you have planned accordingly, but you just want to ensure more worthwhile results? That’s where ly – err, deception – comes in. Two types of deception are commonly used by researchers to gather results with a greater degree of validity: active deception – in which participants are misinformed about certain aspects of a study, such as its true purpose – and passive deception when they are not made aware of certain aspects of a study.
It is often necessary to deceive users during research because giving participants complete information will likely change how they view what they’re doing, how they think, what they do and what they say. In turn the results are less valid. Robert Kerr provided a good example of this
back in March known as “the Good Subject”; a respondent who – upon knowing the true purpose of the study – will be eager to say and do the things they think the experimenter wants, rather than what they would do naturally.
Anything we can do to uncover more valid results is a step in the right direction. To that end, here are a number of lies that you can use to obtain more valid results.
Tell them you had nothing to do with the project
“I’ve not worked on this at all so please feel you can be honest in your opinions”
Telling the participant you designed the thing they’re testing will very likely ruin the validity of the research. Non-confronters, people-pleasers, and aforementioned Good Subjects tend to go out of their way to avoid conflict and will therefore refrain from making negative remarks. Instead, they’ll be full of overwhelming praise even if they noticeably struggle on many of the tasks.
Even if you are the person who designed the product being tested, just omit that information. If they ask, lie. Say you’re not part of the design team at all; you’re just “a researcher.” In fact, even if they don’t ask, you’re better off denying any affiliation with the software whatsoever – they’re probably thinking it.
“I’m actually not familiar with this software so I’m afraid I can’t help you. Would you mind spending another minute on this task whilst talking me through your thoughts and expectations?”
Even if you deny having designed the product that they’re about to use, respondents will likely assume you know the product they’re made to use. If you’re asked to help, use your judgement and assess the length of time the participant has already spent on the task. If they haven’t tried long enough – and they’re not overly stressed – play dumb. This can often instantly refocus them. Another option is to state that you would be “unable to help them as in real life;” however, this implies that you do know how to complete it which can add to their frustration and performance pressure.
Still not sure what “playing dumb” is? The user may ask you “What should I press here?” To which you might say “What would you expect to press?” This is a good start, but you might increase the power of your response (in addition to switching the responsibility of the task back to the user) by adding, “I’m actually not familiar with this software so I’m afraid I can’t help you. Would you mind spending another minute on this task whilst talking me through your thoughts and expectations?”
Lie about the purpose of the study
“We’re just making sure that everything works as you’d expect it to”
By telling the user the true purpose of the study you risk contaminating the results. Research respondents will likely pay more attention and put more focus on any task they know you’re analyzing. This isn’t how they would normally interact with what you’re testing, of course. To keep their reaction as realistic as possible, it’s useful to lie about the purpose of your study.
Lying about a study’s purpose is one of the oldest tricks in the book, according to
Allan Kimmel. His 2001 research paper found it to be one of the most common practices amongst seasoned researchers. Though it’s easy – even natural – to do, be sure you tell the truth after the the test has concluded. More on this later.
Lie about the number of people observing the test
“One or two people might pop into the room next door to watch for a bit, is that ok?”
User research sometimes takes place in a room with a two-way mirror so that the researcher’s client(s) can observe the test in an adjoining room. If there are lots of people behind the two-way mirror observing, don’t let the user know this or it will put them under immense pressure. When respondents know they’re being watched, they often feel pressure to say positive things and perform well.
Lie about how well they’re doing
“Oh, fantastic; that’s really useful!”
Speaking of performing well, some users – especially first timers or shy respondents – may need the occasional bit of reassurance and/or encouragement. A good example would be “Oh, fantastic; that’s really useful!” (even if it isn’t) whilst keeping your body language fairly neutral. This phrase can also be used to gain more comments from the user and can be very effective at helping users to feel more comfortable expressing what they dislike.
Lavishing praise might not seem like a lie per se, but it’s just as powerful. Give it sparingly. Overly positive reinforcement can actually encourage a very specific response from the user, leading to confirmation bias.
One important caveat
Okay, you understand the notion of research validity and you’ve got a bevy of lies you just can’t wait to tell. What’s the catch? Although lying can help you get more valid results, it’s very important that you don’t impinge on the ethical guidelines set by the APA (American Psychological Association):
- Any deception must be justified in terms of significant scientific, educational or applied value that outweigh any risks to participants.
- It must not cause physical pain or emotional distress.
- The researcher must debrief the participant at the end of the session.
These ethical guidelines particularly apply to Lie #3. Always explain the true purpose of a study at the end of the research session. This should be done carefully to ensure the user is clear of the importance of the lie(s) and how telling the truth would have likely changed their response. It’s a good chance for them to further reflect and you may find that at this point, when the participant is relaxed because in their mind the research is over, some of the most useful insights can be gleaned.
Remember these are white lies that aren’t intended to harm the participant in any way. Using them can help participants feel at ease which encourages more honest responses and therefore higher research validity. You don’t need to use them all the time.
Before you put them through their paces, carefully consider the aim of your research and the impact of each lie. You may wish to just test out 1 or 2 lies until you feel comfortable. It may feel odd at first, but remember it’s for the benefit of both the user (to put them at ease) and the research (to gain better results) as a whole. When you notice the user’s posture or facial expression visibly relax, you’ll know the lie has worked well.
Are you ready to start lying to get more from your research? It’ll be our little secret!