Five Tips for Conducting Scientific Research in the UX World

Despite the fact that research plays such a pivotal role in the practice of user-centered design, much has been written about how to approach it in a “quick and dirty” manner. Why the rush? I believe that the application of a more rigorous, scientific methodology could lend some much-needed credibility to our approach.

My love story with research began almost a decade ago. One day, while working as a novice prototyper, I was instructed to get feedback from customers. So — awkwardly — I introduced my ideas to potential users. Some told me what they liked; others gently glossed over what they would improve. I came away feeling accomplished.

Little did I know. My subsequent training as a scientific researcher helped me see the error of my ways. I realized that, in that moment, I used biased responses to inform my design. I heard what I wanted and not necessarily what I needed.

A rigorous approach to research provides a much clearer path to unbiased findings, findings that go a long way towards informing our design. This article covers five perspectives to that end. Starting with research plans, we’ll cover details of testing methodologies and even the role of the researcher herself. Finally, we’ll discuss the ways these tips apply to our research in practice.

Go back to where it all began

All scientific research projects begin with a research plan, a document that outlines:

• The problem (or the research questions) to be explored,
• A summary of existing literature,
• The hypothesis(es) or an extrapolation of any patterns evident in the existing literature,
• The research participants who will take part (more on this, below),
• The data collection methodology(ies) to be employed,
• The planned analysis methods, and
• Any expected results.

The goal in writing a research plan is to be absolutely certain that the entire team understands not only the purpose of the study but also the fact that each aspect of the study has been given due consideration.

Developing a sound research plan requires that we begin with an extensive review of existing theories, models, and other research studies. This ensures that we aren’ t reinventing the wheel. For instance, if the study is based around the System Usability Scale, the best thing to do is to read the original paper to truly understand the scale. Finding original research is more valuable than pretty diagrams or the popularity of the source. Valuable academic citation sites include Google scholar and Microsoft Academic Search. While there’ s always the risk of playing a game of “telephone”, these documents often go through extensive committee review which minimizes the chance that they will contain incorrect information.

Determine the number of participants beforehand

Sample size has been a hot topic for a while now. Some researchers assert that five participants will suffice²; others calculate their sample size based on the power that they want to achieve³; still others believe that a higher number has a lower percentage of risk associated with it⁴. My take is that the sample size depends on the methodology of the study.

For example, a qualitative, exploratory study on mobile phone usage behavior needs descriptive, insightful data, so the number of participants depends on the richness of the information received. But, a quantitative study, such as looking at the effects of mobile phone usage on behavior depends on confidence limits and intervals as well as analysis methods. The more analytical you want to be, the bigger your sample size needs to be.

Either way, the key is to determine the number of participants before conducting our research and to continue researching until we’ ve hit that number. This ensures that we aren’ t swayed by early trends that might ultimately cause us to miss subtle issues. The Three Mile High tragedy is a painful reminder of the severity of subtle issues.

Don’t let your interests conflict

Scientific research focuses on objectivity. For that reason, it always begins with getting approval from an Institutional Review Board (IRB), an academic organization that approves and monitors any research involving humans. The IRB requires that all researchers state they do not have a conflict of interest in the research project at hand.

So, what does this imply for UX designers? Simple: designers shouldn’t research their own designs.

Designers inevitably design things that make sense to themselves. This is beneficial in some ways, but it also paves the way for hundreds of biases to affect decision making. In order to gather unbiased research to inform designs, a trained, unbiased researcher needs to have the final say on the questions, and decipher the answers. This helps avoid experimenter biases like interpretive bias and observer bias.

Test the test

Pilot tests are tests of a research plan. For scientific researchers, pilot tests are necessary in order to ensure the validity of the research plan and help identify possible problems with it⁵. Ideally, pilot tests are conducted with a group of users that are representative of the target audience.

The pilot test works exactly like the proposed one, but instead of looking for data, it allows us to catch errors in our test itself. For example, if we are pilot-testing a survey and users don’ t understand the word ldquo; cumbersome” , we might remove that from our final survey. With a survey, we’ ll also time how long users take to complete it, make sure that every question is understood correctly and ask the participants for candid feedback.

If we’ re doing a usability test, we’ ll provide the instructions and watch them complete the tasks that we plan to assign to users, to ensure that our instructions are clear; we’ ll remind users to think aloud and to be frank with their opinions, as they would in an actual test; and, most important, we’ ll take notes every time they ask that a question be repeated or for more clarity.

Make sure to stick to the planned script and behave as though this was a regular research study. Ask for honest feedback on how users would improve the overall study and let your expertise as a researcher apply their answers accordingly.

Typically, results of a pilot test are only used to modify the actual test. Results like answers to surveys, time taken to complete tasks, etc. should not be incorporated into the final results of the research to ensure consistency.

De-bias, de-stress, de-tect

Scientific research often requires extensive vetting of researchers — the people conducting the research — prior to their participation in a project. The latest trend in the UX world is to get everyone involved with the research. As a researcher, nothing excites me more than this but that being said, it is extremely important to acknowledge that the inexperience of a researcher and its number of open (versus hidden) observers can be inversely proportionate to its overall ldquo; success.”

For instance, usability testing (arguably, the most common type of research method in the UX world), can be extremely stressful for participants⁶. Aside from being asked to ‘perform’ , users are sometimes put in unnatural conditions which can be very nerve wracking. This, in turn, could hinder performance and risk invalidating our findings.

Another thing that affects performance is the fact that participants change their behaviour when they know they’ re being observed, something otherwise known as the Hawthorne effect. Worse still, this effect is only exacerbated as the number of observers increases. So while it’ s definitely good to get more people involved and invested in research, there are a few precautions we should take in order to minimize their potential negative effects.

1. First, whenever we’ ve got a group of people involved in the research process, we should always ensure the facilitator of a research session has some experience and training so that they’ re not unknowingly influencing participants. Keep an eye out for leading questions and analyze the results accordingly.
2. Second, either keep the observers hidden or to a minimum. A researcher’ s main job is to keep the data as pure as possible (objectivity, remember?), and a stressed participant does not provide reliable data.
3. Third, let’ s remind the users that we had nothing to do with the design, so that users aren’ t hesitant to give too much negative feedback.
4. Fourth, always remind the user that we’ re testing the product and not them. This is (hopefully) old news, but users need to be reminded of this constantly.
5. Fifth, and finally, always keep an eye out (or a ear, if the session is remote) for any sign of stress. If the participant starts to appear stressed, immediately change the topic or consider ending the session. The key here is to note the difference between a stressful or frustrating design interaction and a stressful research session. The former provides valuable insight while the latter can provide unreliable data.

Repurposing the scientific method

In summary, I suggest taking a leaf out of the scientific researchers ’ book:

1. Plan your study out and read the sources to get accurate information.
2. Choose a number of participants before testing, and stick to it regardless of the first trends that appear.
3. Be alert! Watch out for bias when conducting research.
4. Test our tests.
5. Avoid biases, stress, and leading questions.

Most importantly, don’t shy away from rigor in the research process; it’ s the only thing that can lead to truly dependable results!

References

1. Fin, P. (2006). Bias and Blinding: Self-Fulfilling Prophecies and Intentional Ignorance
2. Nielsen, J. (1993). Usability Engineering. Boston :AP Professional.
3. Kraemer, C. & Thiemann, S. (1987). How many subjects? Statistical power analysis in research. Oaks, CA, US: Sage Publications, Inc. (1987). 119 pp.
4. Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increase sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 2003, 35(3), 379-383.
5. van Teijlingen, E. & Hundley, V. (2001). The importance of pilot studies. Social Research Update. Issue 35.
6. Schrier, J. (1992). Reducing Stress Associated with Participating in a Usability Test. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 1992 36: 1210. DOI: 10.1177/154193129203601606.