5 Useful Lies to Tell User Research Participants

Although design researchers vary in their methodology, no one argues with valid results. Lisa Duddington explains how to introduce both passive and active deception into the research process.

If you’ve ever run a research or usability test, you’ll know they can be tricky to facilitate. After all, you’re dealing with people; and people come with a whole host of existing preconceptions, personalities, emotions, and experiences. One thing that can help you to gain more honest and thereby useful feedback from research participants is, in fact, to lie to them.

Data is a sorted sort. Not only must it be properly contextualized and analyzed in order to bear useful information, it must also be collected and collated in a prudent fashion to begin with. Researchers go through this high level of detail to ensure the validity of their results. Dr. Marion Joppe of Ryerson University provides a more exacting definition:

Validity determines whether the research truly measures that which it was intended to measure or how truthful the research results are.

User researchers can increase the validity of their results in a variety of ways. Sometimes they conduct research “on-the-road” – known as
ethnographic research
– to interact with participants in their context of use. Other researchers go as far as recreating the environmental setting in which the product will be used. For example, if testing a television or video game, they might rearrange their lab to feel like a living room (comfy sofa, pictures on the wall, etc). If the product being tested is something that’s mostly used in the evenings, they might change the lighting in the room. If participants would often be interrupted while doing a particular task, the researchers might frequently interrupt their participants during the test. You get the idea.

In his 1994 paper Usability: Practical Methods for Testing and Improvement, Miles Macleod posited the following questions to aid the validity of research:

  • Are you looking at the right things to be representative of real-world use?
  • Are you collecting the right data and the right amounts of it?
  • Are you analyzing the raw data reliably?

The general consensus across these approaches – and Macleod’s questions – is that to increase the validity of a test requires scrutiny and planning. Though we’re all aware that planning is a good thing, what if you have planned accordingly, but you just want to ensure more worthwhile results? That’s where ly – err, deception – comes in. Two types of deception are commonly used by researchers to gather results with a greater degree of validity: active deception – in which participants are misinformed about certain aspects of a study, such as its true purpose – and passive deception when they are not made aware of certain aspects of a study.

It is often necessary to deceive users during research because giving participants complete information will likely change how they view what they’re doing, how they think, what they do and what they say. In turn the results are less valid. Robert Kerr provided a good example of this
back in March
known as “the Good Subject”; a respondent who – upon knowing the true purpose of the study – will be eager to say and do the things they think the experimenter wants, rather than what they would do naturally.

Anything we can do to uncover more valid results is a step in the right direction. To that end, here are a number of lies that you can use to obtain more valid results.

Tell them you had nothing to do with the project

“I’ve not worked on this at all so please feel you can be honest in your opinions”

Telling the participant you designed the thing they’re testing will very likely ruin the validity of the research. Non-confronters, people-pleasers, and aforementioned Good Subjects tend to go out of their way to avoid conflict and will therefore refrain from making negative remarks. Instead, they’ll be full of overwhelming praise even if they noticeably struggle on many of the tasks.

Even if you are the person who designed the product being tested, just omit that information. If they ask, lie. Say you’re not part of the design team at all; you’re just “a researcher.” In fact, even if they don’t ask, you’re better off denying any affiliation with the software whatsoever – they’re probably thinking it.

Play dumb

“I’m actually not familiar with this software so I’m afraid I can’t help you. Would you mind spending another minute on this task whilst talking me through your thoughts and expectations?”

Even if you deny having designed the product that they’re about to use, respondents will likely assume you know the product they’re made to use. If you’re asked to help, use your judgement and assess the length of time the participant has already spent on the task. If they haven’t tried long enough – and they’re not overly stressed – play dumb. This can often instantly refocus them. Another option is to state that you would be “unable to help them as in real life;” however, this implies that you do know how to complete it which can add to their frustration and performance pressure.

Still not sure what “playing dumb” is? The user may ask you “What should I press here?” To which you might say “What would you expect to press?” This is a good start, but you might increase the power of your response (in addition to switching the responsibility of the task back to the user) by adding, “I’m actually not familiar with this software so I’m afraid I can’t help you. Would you mind spending another minute on this task whilst talking me through your thoughts and expectations?”

Lie about the purpose of the study

“We’re just making sure that everything works as you’d expect it to”

By telling the user the true purpose of the study you risk contaminating the results. Research respondents will likely pay more attention and put more focus on any task they know you’re analyzing. This isn’t how they would normally interact with what you’re testing, of course. To keep their reaction as realistic as possible, it’s useful to lie about the purpose of your study.

Lying about a study’s purpose is one of the oldest tricks in the book, according to
Allan Kimmel
. His 2001 research paper found it to be one of the most common practices amongst seasoned researchers. Though it’s easy – even natural – to do, be sure you tell the truth after the the test has concluded. More on this later.

Lie about the number of people observing the test

“One or two people might pop into the room next door to watch for a bit, is that ok?”

User research sometimes takes place in a room with a two-way mirror so that the researcher’s client(s) can observe the test in an adjoining room. If there are lots of people behind the two-way mirror observing, don’t let the user know this or it will put them under immense pressure. When respondents know they’re being watched, they often feel pressure to say positive things and perform well.

Lie about how well they’re doing

“Oh, fantastic; that’s really useful!”

Speaking of performing well, some users – especially first timers or shy respondents – may need the occasional bit of reassurance and/or encouragement. A good example would be “Oh, fantastic; that’s really useful!” (even if it isn’t) whilst keeping your body language fairly neutral. This phrase can also be used to gain more comments from the user and can be very effective at helping users to feel more comfortable expressing what they dislike.

Lavishing praise might not seem like a lie per se, but it’s just as powerful. Give it sparingly. Overly positive reinforcement can actually encourage a very specific response from the user, leading to confirmation bias.

One important caveat

Okay, you understand the notion of research validity and you’ve got a bevy of lies you just can’t wait to tell. What’s the catch? Although lying can help you get more valid results, it’s very important that you don’t impinge on the ethical guidelines set by the APA (American Psychological Association):

  • Any deception must be justified in terms of significant scientific, educational or applied value that outweigh any risks to participants.
  • It must not cause physical pain or emotional distress.
  • The researcher must debrief the participant at the end of the session.

These ethical guidelines particularly apply to Lie #3. Always explain the true purpose of a study at the end of the research session. This should be done carefully to ensure the user is clear of the importance of the lie(s) and how telling the truth would have likely changed their response. It’s a good chance for them to further reflect and you may find that at this point, when the participant is relaxed because in their mind the research is over, some of the most useful insights can be gleaned.

Roundup

Remember these are white lies that aren’t intended to harm the participant in any way. Using them can help participants feel at ease which encourages more honest responses and therefore higher research validity. You don’t need to use them all the time.

Before you put them through their paces, carefully consider the aim of your research and the impact of each lie. You may wish to just test out 1 or 2 lies until you feel comfortable. It may feel odd at first, but remember it’s for the benefit of both the user (to put them at ease) and the research (to gain better results) as a whole. When you notice the user’s posture or facial expression visibly relax, you’ll know the lie has worked well.

Are you ready to start lying to get more from your research? It’ll be our little secret!

About the Author

Lisa Duddington

Lisa runs the award-winning UX Agency Keepitusable. She is an expert user researcher, who enjoys improving the design of things like websites, mobiles, software and products to make both people and businesses happier and more successful. You can follow Lisa on Twitter @usabilitygal and read more of her articles at her usability blog and keepitusable blog.

Related Articles

15 Comments

  • Colleen Reply

    Hi Lisa,

    Just wondering what your opinion is about providing a Statement of Informed Consent to testers? Seems like if there is active deception involved during the process, it would have to be declared in that statement upfront…Thanks.

    • Lisa Duddington Reply

      Hi Colleen

      That’s a good point. The problem with doing that is it may make the participant more wary, conscious of what they’re doing and also lessen their trust with you as the researcher. You then get back into the problem of less valid results… It really depends on the kind of deception involved and the potential it has to harm the user. For the majority of UX research you’ll be fine declaring any deception at the end of the session.

      Thanks for your comment Colleen.
      Lisa

  • David Reply

    “We’re just making sure that everything works as you’d expect it to” — How is this a lie when it comes to the purpose of the study? Isn’t that the point of usability testing? If it is a lie, what are you actually trying to find out?

    • Lisa Duddington Reply

      For usability testing you’re right it wouldn’t be a lie David. This article is for user research (as well as usability testing) which can have a broader range of aims. For example, I’ve conducted user research where usability was actually further down the list of priorities. You could also use this lie to misdirect the user – to give them the impression you are analysing the software, when in fact you may be primarily interested in their physical interaction with it. Are they using their left/right hand or both? How are they holding the device? Are they using the pad/tip/nail of their finger? This can give you vital input into the design of both software and hardware.

  • frances Reply

    I’ve done work practice, user analysis for 20 years.
    Everything you want to accomplish with these lies can be accomplished without them. Why lie?

    • Rinchan Reply

      lies will work in few scenarios but I completely agree with Frances that Why lie when we can do the job without them.

    • Lisa Duddington Reply

      I disagree, there are many scenarios where telling the whole truth would change the user’s perception of the study, would bias their responses and even change them completely. I have witnessed this many times and it can be avoided with a few white lies or omission of certain facts.

  • Joe Reply

    It’s a really interesting topic (I’m a sucker for anything ethics related). I have to say I think the most important part is the debrief after; I’ve had to lie in certain pieces of research about who the testing is for, but afterwards explained why I did it. Everyone has been completely understanding. These aren’t lies that will upset the participant in any way, and as you say, may help them feel more comfortable throughout – and ultimately help get the best out of the session for everyone concerned

  • Jo Anders Reply

    I don’t agree with the premise that you have to lie, or that ‘deception is okay’, so I’m with Frances too.
    I do agree with the dangers and pitfalls described, but there are different ways to achieve good results that don’t include dishonesty or deception. -The article mentions some of them:
    Not mentioning or omitting information is fine, as well as asking ‘what do you think it does?’ when people ask about a button.
    Limiting the level of detail, like “Some people will be observing from the next room” is also good without it being a lie. As for purpose, I don’t think you need to be very specific at all; Just tell people that you want them to try something on the computer or the general website in question in order to get their feedback, and they’ll be okay with that.
    Also, your job as a facilitator is to establish trust and relaxed rapport, and I think actual lying (“I’ve not worked on this at all..”) undermines this effort and creates unnessecary distance.
    As for ‘the good user’ I find that encouraging people to be themselves and saying that it’s totally ok and even useful if they’re critical, works just fine.
    And less savvy and ignorant users, remember that they represent a range of users too.
    I wouldn’t feel good about praising someone totally clueless, but I do assure them that their user behavior is quite common, that they’re normal and not alone – which is almost never a lie ;)

    • Lisa Duddington Reply

      Thanks Jo. Some really useful suggestions and constructive comments there. I enjoyed reading them and have used many myself. Of course, I’m not saying you have to lie at all, these are simply a tool available that can be really useful in certain circumstances :)

  • Jess Reply

    It feels like some people are a bit unnecessarily squeamish about lying to research participants. Some of the most interesting pieces of psychological research – some of which guide our design work – have involved experiments where the situation was carefully manipulated to keep participants’ behaviour authentic. Researchers often tell participants that the purpose of the study is totally different to the real goal, or even pose as fellow participants themselves. A willingness to be kept in the dark – at least for the duration of the experiment – comes with the territory of putting yourself forward as a research participant. Medical test wouldn’t be much good if participants had to be told when they were being given a placebo, would they?

    Clients call on professional UX researchers on the understanding that they’ll use their skills to design and execute an experiment that will provide the most useful, valuable insights. I’d say it would be more of an ethical problem to tarnish the validity of the experiment than it would be to lie to participants.

  • Mike Pearce Reply

    In the UK, ethics laid down by the BPS only apply to scientific studies. I’m pretty sure it’s the same in the US (for example, many “reality” TV shows would be unethical as psychological studies, the US have far crazier ones than the UK). Unless you’re using the data you’ve gathered to write a paper which will then be peer reviewed, it’s unlikely any psychological body will give a hoot.

    As for lying, I agree with Jess. If you lie (and they seem like sensible, pragmatic lies), then tell the truth afterwards, you risk tarnishing the ethics of your entire practice (“don’t go there, they lie to you!”) and that participant will ALWAYS expect you to lie in the future.

    However, there are probably ways around all the above lies, as an example, you could design tasks to eliminate the “good subject” problem by changing the demand characteristics of the experiment to mask the real expected outcome.

  • Andy Feliciotti Reply

    Love the point about avoiding affiliation with the project, I never though to do that

  • Eric Reply

    I’m in late – two weeks after the post – but timing is right for me. Preparing my first-ever test where I’m going to mislead about my identity (for the bulk of the test). Different testing scenario and objectives than we’ve used previously.

    That said, in our usual scenarios when we can’t hide the company for which we work, we’ve gone the extreme opposite and said, “Yes, we’re involved with this product, but we need it to work for you, not for us. We’re grown-ups, so please be brutally honest and don’t worry about our feelings. Think of yourself as speaking up on behalf of your neighbors, family and friends who also are our customers. If it doesn’t work for you and them, then we need to change it.”

    We’ve received very frank answers. Sometimes they ask, “You want me to be honest, right?” and we say absolutely. We’ve then heard things like, “I get what you’re trying to do but this just isn’t working for me.” Then we get into the why and go from there.

  • Katharina Reply

    I especially disagree with No.5. First of all, if you say “oh that is useful” but you don´t mean it, people can pick that up. Not all, but some will. Second, why do you judge already in the middle of the test that something is useful or not? Shouldn´t we just take all the information that we get and then take it to analysis stage?

Leave a Comment on This Article