Interview with the Authors of Remote Research

February 16th, 2010
Published on
February 16th, 2010

Research is what Bolt | Peters does best. As a digital research firm, their job is to uncover and determine the ways in which real users use real products with their “natural” environments. In order to achieve the most accurate data possible, researchers at Bolt | Peters employ a broad variety of research methods. One method in particular, known as Remote Research, has garnered considerable attention.

The book cover for Remote Research, published by Rosenfeld Media.

In their forthcoming new book, authors Nate Bolt and Tony Tulathimutte address the questions they receive while practicing Remote Research. From the mundane, “what is remote research,” to the more intriguing “how does it compare with more conventional design research practices?” their book promises to cover it all.

In an odd turn of events, we at the UX Booth found ourselves with the opportunity to sit down with these esteemed researchers and give them a dose of their own medicine, asking questions of our own. In the following interview, learn what design research means to Nate and Tony and even how you can get started. If that’s not enough incentive to employ research within your own design practice, we’re even giving away some copies of the soon-to-be-released book. Enjoy!

I would hazard a guess that most of our readers who are interested in experience design (and remote research), aren’t in a position or at a company where it’s treated as a first-class citizen. Although the question is frequently posed, I have to ask: how do you suggest these people introduce remote research into their product-development process?

Add a couple of informal UX research sessions into your next design project, whether or not design research is specifically included in the budget or timeline. Don’t ask anyone if they think it’s a good idea, just do it. It only takes 2-3 hours, and costs nothing. All the standard remote UX tools have free trials, and you can grab a screen sharing tool like GoToMeeting or Acrobat Connect, a screen recording program like Quicktime 10 or iShowUHD, and you’re good to go. Then edit the recording in iMovie or another free video editor. You can also use an in-person tool like Silverback.

When you show photos, user quotes, or quick video highlights to your stakeholders, it will blow them away if they weren’t expecting it, and it can all be done in roughly the amount of time we each spend on Facebook and Twitter every day.

Can you briefly explain the idea behind time-aware research and “live recruiting?”

In traditional user research, you ask someone in advance to come in to a boring usability lab and pretend to care about the interface or design you’re researching. With time-aware research, you intercept them live with a web tool like ethnio or a mobile-based form, and ask them to participate in a phone interview just as they’re about to use an interface, on their own initiative and timeline. What’s important is that you’re capturing someone right at the moment when they care about what they’re doing, whether it’s making a donation, buying a phone, purchasing enterprise software, etc. The criteria by which they make click decisions are totally different when they really need to book travel that minute.

A great example of a time-aware research project is the popular Track Your Happiness project at Harvard, which uses SMS messages to intercept people and answer questions about their happiness into a mobile form. The happiness data is reported in “real-time,” which provides far more accurate data than people filling out some survey about how happy they were last month. Basically, it eliminates our misrepresentations about interactions by peeking into real-life behavior and time.

Do you find that you obtain better results presenting/observing reactions to early sketches or wireframes vs. functional prototypes or fully-functioning applications/products?

The answer to this question is actually a funny comment on the current state of the web. We’ve tested 2,300 participants over ten years, with at least 40% of those sessions involving a prototype, a scanned napkin sketch, static JPGs, or wireframes. We never use the word “prototype,” and almost always, after we redirect people to a static wireframe, they say “yeah, this looks like other sites I’ve used.” In other words, even with a crap sketch, as long as it clicks through to another crap sketch, people can relate that to their normal browsing experience. That’s how crappy parts of the web are. People aren’t shocked by a black and white version where only one thing works.

That said, the more functional interactions our participants have, the more insight we get, so the results do get incrementally better with more fidelity, but testing sketches is still incredibly useful.

In today’s web application space, two terms are hot topics: “agile development,” and “user experience design.” How do you see research (in general) keeping step in a fast-paced, agile development environment? Applications that are in-development simply don’t have the users with which to conduct time-aware research, so I wanted to ask for your thoughts on how prudent experience/interaction designers can proceed in this “pressure-cooker” situation. (I’m aware of—and eagerly anticipate reading—Anders Ramsay’s book on Agile Experience Design)

Go guerilla. Andy Budd from ClearLeft loves this approach and so do we. The philosophy is: instead of forgoing testing, or trying to get it approved, just grab someone whose paycheck doesn’t involve your design or development, and watch them interact. The same goes for remote testing—you don’t even have to leave your chair to observe someone’s functional experience with an interface.

We did a recent study for, who don’t practice Agile but do release very rapidly, and they made changes on the fly based on what the designers and developers were seeing in remote testing. We conducted it from a conference room where it was easy for their team to come observe, minimizing the gap between developers and designers and “the research.” They just saw UX issues and addressed them.

Research is seen primarily as an academic pursuit: how did you get started and how do you suggest others do the same?

Everyone here at Bolt | Peters studied some variation of Cognitive Science, Linguistics, or Human-Computer Interaction. I worked as a design intern pushing pixels for a web development company in 1998 called Clear Ink, and I convinced them to let me pitch usability to their clients, since everyone was listening to punk 21 yo. kids back then. Eventually, one of the clients thought it sounded like a good idea and our first study yielded designs that got a 150% conversion rate increase and millions of dollars more in web sales. Then it was off and running.

So the steps I’d suggest are:

  • Take a class or two in a related field.
  • Intern somewhere that conducts UX and IxD.
  • Just do research without asking.
  • Measure results and highlight the awesomeness.

That’s all, folks!

Thanks again to Nate and Tony for taking the time to answer our questions and share their thoughts with the community. Remote Research is an exciting, emerging methodology which increasingly appeals to contemporary design endeavors.

From what I’ve read thus far, it’s going to be an awesome book. So good, in fact, I would recommend purchasing two copies: one to keep and one to share—which, come to think of it …

Want to win a copy?

In case you hadn’t noticed, Rosenfeld Media is one of the primary sponsors of UX Booth this quarter. As a consequence, we’ve got 3 copies of Remote Research to give away to our community. To enter, simply leave a comment below and make sure to include your twitter ID. Next, be sure to follow us (@UXBooth) on twitter. If you’re selected, we’ll be in touch via a Twitter DM to obtain your mailing address. Thanks, and good luck!

Rather just purchase one?

Readers of UX Booth are entitled to a 15% discount off any of the fine products found at Rosenfeld Media. Simply enter code UXBOOTH at checkout.

About the Authors

Nate Bolt

Nate is fascinated by the personal, social, and cultural role of technology, and how research and design can transform those roles. After pioneering and directing the User Experience department at Clear Ink in 1999, which included the construction of Natural Environment and Remote Observation laboratories, Nate co-founded Bolt | Peters. He now serves as el presidente, where he has overseen hundreds of user research studies for Sony, Oracle, HP, Greenpeace, Electronic Arts, and others. Beginning in 2003, he led the creation of the first moderated remote user research software, Ethnio, which is being used around the world to recruit hundreds of thousands of live participants for research.

Tony Tulathimutte

Tony specializes in user research for video games, working for Time Warner’s GameTap service and NHN USA’s online games portal, (His honors thesis was about video game controllers, and his Master’s thesis concerned game menu interfaces; in this way, he makes the world a better place.) He joined Bolt | Peters in 2007, and has done remote studies for Autodesk, AAA, HP, and Harvard Business Review; more recently, he was the lead researcher on the player experience study for EA’s Spore.

Additional Resources

Research Super Guide
Complete Beginner's Guide to UX Research

UX research - or as it’s sometimes called, design research - informs our work, improves our understanding, and validates our decisions in the design process. In this Complete Beginner's Guide, readers will get a head start on how to use design research techniques in their work, and improve experiences for all users.