The Art of Guerilla Usability Testing

How can we go about testing our designs with minimum effort? David Simon explores guerrilla usability testing: the art of quickly soliciting – and later analyzing – user feedback.

Guerrilla usability testing is a powerful technique. Designer Martin Belam describes it as “the art of pouncing on lone people in cafes and public spaces, [then] quickly filming them whilst they use a website for a couple of minutes.” Let’s skip the pouncing part and instead focus on its subtleties, including how to obtain and share feedback with our team.

I recently worked on a quickstart project in which my team was asked to build a responsive website in a short amount of time. We were given very little time to code (let alone conduct research) for the endeavor, yet by employing guerilla usability testing along the way we collected feedback on the brand position. Eventually, we aligned our designs to both customer expectations and business goals.

Once a week throughout the project, we tested different kinds of prototypes to bring the business’s ideas to life. For example, while mid-development, we sketched a mobile version of the site on index cards and did a quick assessment. This revealed navigational problems (which guided us to rethink a key point in the customer journey) and even ended up shaping a bit of the brand’s media material. What’s more, guerilla usability testing opened our stakeholders’ eyes so that they challenged their own, innate assumptions about “the user.”

We iterated through our design ideas using lo-fi techniques like paper prototyping. Sketch by Chris Cheshire.

The bottom line? Guerilla usability testing presented itself as an easy-to-perform technique for refining the user experience. It helped us validate (and invalidate) critical assumptions at cheap cost and with rapid speed.

Breaking it down

It’s hard to see the magic that guerrilla usability testing affords and not want in on the action, right? Here are some basic questions to consider before getting started:

  1. What shall we test?
  2. Where will we test?
  3. With whom will we test? and, of course,
  4. How will we test?

What shall we test?

One of the best parts about this kind of testing is that it can be done with almost anything, from concepts drawn on the back of napkins to fully functioning prototypes. Steve Krug recommends testing things earlier than we think we should and I agree – get out of the building as soon as possible.

Test what the product could be so as to shape what the product should be. Even loosely defined UI sketches can be a great way to evaluate a future product. In fact, recent research shows that lower-fidelity prototypes can be more valuable concerning both high and low-level user interactions.

Where do we test?

Where we conduct tests affects how we perform and document our work. For instance, if we’re testing a new mobile app for a retail chain, we might go to the store itself and walk the aisles; if we’re working on “general” office software, we might test it with coworkers in a different part of the office; etc. The point is: let context drive the work.

With whom do we test?

When designing for the mass market, it’s easy enough to ask friendly looking strangers if they have a couple minutes to spare. Public spaces and shopping centers present some of the best places to do this on account of the sheer amount of foot traffic they receive (as well the relaxed nature of the environment). With more specific user sets, however, it’s useful to target subjects based on their context (see above): a mixture of location and behavior.

Coffeeshops are great because you’ll often find test subjects from varying cultural backgrounds and different age ranges.

How do we test?

Testing is fairly straightforward: have participants talk aloud as they perform tasks. Use the think-aloud protocol to test overall product comprehension rather than basic task completion. The key is to watch customers fiddle with a product and silently evaluate its usability. As Sarah Harrison explains, “Observing users is like flossing–people know they’re supposed to do it every day, but they don’t. So just do it. It’s not a big deal.”

Always start with open-ended, non-leading questions like:

  1. What do you make of this?
  2. What would you do here?
  3. How would you do [that]?

By answering these kinds of questions, participants tell a loose story in which they explain how they perceive a product. Along the way, we can generate ideas for how to improve things in the next iteration.

Employing the technique

Guerrilla usability testing is very much about adapting to the situation. That said, here are some helpful hints that I find consistently work in different international contexts:

  1. Beware the implicit bias. While coffeeshops are a great place to find test participants, focusing on people who frequent them introduces bias to our work. Simply acknowledging this implicit bias can help designers neutralise subjective experiences and account for individual differences. Remember to target different genders and be fair in who you approach.
  2. Explain what’s going on. Designers should be honest about who we are, why we’re testing, and what sort of feedback we’re looking to receive. Oftentimes, it’s best to do this with a release form, so that people are fully aware of the implications of their participation – like if it’s going to just be used internally versus shared globally at conferences. These sort of release forms, while tedious to carry around, help establish trust.
  3. Be ethical. Of course, being honest doesn’t mean we need to be fully transparent. Sometimes it’s useful to skip certain information, like if we worked on the product they’re testing. Alternatively, we might tell white lies about the purpose of a study. Just make sure to always tell the truth at the end of each session: trust is essential to successful collaboration.
  4. Make it casual. Lighten up tests by offering cups of coffee and/or meals in exchange for people’s time. Standing in line or ordering with a test subject is a great opportunity to ask questions about their lifestyle and get a better feel for how a test might go.
  5. Be participatory. Break down barriers by getting people involved: ask them to draw – on a napkin or piece of notebook paper, for example – what they might expect to see on the third or fourth screen of a UI flow. This doesn’t have to be a full-blown user interface necessarily, just a rough concept of what’s in their head. You never know what you’ll learn by fostering imagination.
  6. Don’t lead participants. When you sense confusion, ask people what’s going through their head. Open them up by prodding, saying “I don’t know. What do you think?”. People in testing situations often can feel as though they are being tested (as opposed to the product itself), and therefore can start to apologise or shut down.
  7. Keep your eyes peeled. It’s important to encapsulate passing thoughts for later analysis. Ethnographic observation is one good way to capture what you were thinking of during tests. Don’t get too hung up about formalised notes though, most of the time your scribbles will work just fine. It’s about triggering memories, not showing it off at an academic conference.
  8. Capture the feedback. A key part of any testing process is capturing what we’ve learned. While the way in which we do this is definitely a personal choice, there are a few preferred tools available: apps like Silverback or UX Recorder collect screen activity along with a test subject’s facial reaction. Other researchers build their own mobile rigs. The important part to remember here is to use tools that fit your future sharing needs.
  9. Be a timecop. Remember, this isn’t a usability lab with paid users. Be mindful of how much time you spend with test subjects and always remind them that they can leave at any point during the test. The last thing you’d want is a grumpy user skewing your feedback.

Sharing the feedback

Conducting the tests is only half the battle, of course. To deliver compelling and relevant results from guerilla usability tests, designers need to strategically decide how we’ll share our findings with our colleagues.

When analysing and preparing captured feedback, always consider your audience. The best feedback is the kind that understands stakeholders and kickstarts important conversations between them. For example, developers who need to evaluate bugs will have different needs than executives who want to prioritise new features.

Next, when delivering feedback, align it with your audience’s expectations. Try editing clips in iMovie or making slides in PowerPoint. Your co-workers are probably as busy as you, so an edited down “trailer” that highlights relevant results or a bullet-point summary along with powerful quotes is always a good method to keep people listening.

Go guerilla

At the end of the day, guerilla usability testing comes in many forms. There’s no perfection to the art. It is unashamedly and unapologetically impromptu. Consider making up your own approach as you go: learn by doing.

Note: Thanks to Andrew for providing lots of feedback on early drafts of this article.

Related reading

About the Author

David Peter Simon

David Peter Simon is an Experience Designer at ThoughtWorks, a global software company. In his free time, he blogs for Indie Shuffle and Holiday Matinee. Talk with him on Twitter @davidpetersimon.

Related Articles

9 Comments

  • Theresa Neil Reply

    Great tips, thanks for sharing. We’ll add this article in to the Discovery section of UXApprentice.com where we have other resources about quick and dirty ways to get user input early.

  • Adriaan Reply

    On our product we are trying to do exactly the same – I’m very interested to hear more about your approach in getting access to the venue’s of where you did the testing.

    Did you chat to the manager/owners of the coffee shops and cafe’s before you did the testing?

    • David Peter Simon Reply

      Hi Adriaan,

      At cafes, I usually talk to the people behind the counter and let them know my intentions.

      If they’re cool with it, I tend to leave my card (running as a tab) so as to make ordering quick and easy. I also set up my equipment at a table in the back and prepare for the study.

      Good luck with your testing.

  • Manya.pe Reply

    The guerrilla usability testing, can also serve to test how it affects the situation or the place, in the interaction between the user and the website. It would be nice, guerilla usability testing in ecosystems of the product or service.

  • Eddie James Reply

    Really nice breakdown of usability testing in the wild.

    But beware your approach when asking people to participate for free. Many will not be super open when you walk up to them (they may be on a work break or doing work or school work in the coffee shop or be in a hurry or think you are going to try to sell them something). You might get a lot of NOs, which can be discouraging.

    I don’t think asking people to provide feedback for free is a good idea. I’ve found I end up with people with time on their hands and not necessarily the right participants. Plus time is valuable and I think participants deserve something — even if it’s really small.

    I have had success sitting at a table and putting up a sign saying “I’ll buy you coffee for your opinion and 10 minutes of your time.”

    I’ve even paid out of my own pocket. I have no problem spending a few bucks to get good feedback.

    Instead of approaching random people, you could also recruit on craigslist or other online site BEFORE going to to coffee shop. You can have screened users show up at a certain time. Heck, have them all show up at once. This approach may yield better participants and might actually take less time and you don’t have to hope you find the right types of users who just might be in the local coffee shop on the day you visit (which is often unlikely to happen).

    I’ve done the craigslist recruiting (and also on facebook and twitter) in large cities and had people a few blocks away show up within an hour of my posting. So this can be done fast.

    Oh, and if you can’t afford to buy the participants coffee, don’t underestimate the power of t-shirts or other marketing items.

    I’ve seen people spend A LOT of time giving me feedback just to get a crappy t-shirt that the client had boxes of sitting around in a closet somewhere.

    It can even be extra t-shirts from a conference from long ago or all extra large sizes no one wanted. People love t-shirts. I know, it’s strange.

    And finally, if you live near a bus stop or subway station or train station or airport, go there. All the people there are waiting around and often killing time and staring at their smartphones and computers anyway.

    Okay, I’ve given away all my secrets. Now go test something!

  • Øyvin Halfdan Thuv Reply

    Concise writeup!

    Makes it easy to see how much value one can get from a small up-front investment in testing.

    Hope it grabs wide attention!

  • Nascordia Reply

    One thing to really consider when Guerilla Testing is the time and location of when you ‘pounce’.

    Lunch breaks in an urban area can get you rushed, incoherent responses, while saturday breakfast time is generally the most relaxed and productive. Then again, it might be representative of the different times people will use your application.

    One hard demographic to capture is the executive businessman. At Nascordia, we’ve found counter intuitively, that the best place to interact with executives is at the airport, when they’re waiting for a flight.

  • Blog Fikirleri Reply

    Hımm…Great post Peter!

    User satisfaction is always valuable. For the article congratulations.

Leave a Comment on This Article