Home

Tags: user research ux research

The Moderator’s Survival Guide: Handling Common, Tricky, and Sticky Situations in User Research

By Donna Tedesco and Fiona Tranquada

Published: April 7, 2014

This is a sample chapter from Donna Tedesco and Fiona Tranquada’s new book, The Moderator’s Survival Guide: Handling Common, Tricky, and Sticky Situations in User Research. 2014 Morgan Kaufmann.

Moderator’s Survival Guide

Chapter 2: In the Trenches: Six Steps for Handling Situations

Once you’ve understood your role as the moderator and the different styles of moderating, you can start thinking about how to handle the different kinds of situations that may arise during the session. Some of these situations will challenge your interpersonal skills; others will push the limits of your troubleshooting and crisis-handling capabilities. Although there are common circumstances you’re almost guaranteed to encounter, we can’t predict what else might happen. Even after our years in user research, we’re still surprised—and sometimes delighted, shocked, or horrified—by what happens when we’re one-on-one with a participant. This chapter provides a set of guidelines (as shown in Table 2.1) to help you decide exactly what to do when something unexpected, sticky, or tricky happens during your session.

Table 2.1—Step-by-Step Guide for Handling Any Situation

  1. Take a moment to evaluate the situation before jumping to action.

  2. Resolve any threats to physical safety.

  3. Verify that you’re not causing or magnifying the situation.

  4. Check the participant’s comfort level.

  5. Use careful language and tone to probe on the situation and begin to resolve it.

  6. Regain control to bring the session back on track.

2.1 Take a moment to evaluate the situation before jumping to action

You’re running an interview with a remote participant over the phone. The session seems to be going well until the participant doesn’t respond as quickly to your questions. You hear him whisper to someone in the background, but can’t understand what he says. What, if anything, should you do?

When you encounter anything outside of expected session circumstances, take a moment to step back and evaluate the situation. As the moderator, you keep track of so many things that it can be difficult to take that time. But doing so is vitally important. Why?

This moment may feel like a lifetime to you, but it doesn’t necessarily feel that way to the participant! Even in a potentially life-threatening situation—for example, a fire alarm goes off—take a moment to gather as much information as possible before proceeding. Your need to respond appropriately should outweigh your instinct to react immediately. Remember that you’re responsible for the participant’s well-being as well as your own.

If the situation is especially tense or you’re not quite sure what’s going on, you may want to take your moment in another room, away from the participant. Taking a break is an incredibly powerful tool that we’ll discuss more in section 3.7. A break gives you a chance to think about what’s happening in the session without the pressure of monitoring everything else. If you have observers, you can use this break to check with them and get a “gut-check” on what you think is happening and why. The participant may also benefit from a few minutes to cool off, or deal with his distractions.

Table 2.2 shows examples of what you can say if you need to take a break. Note that you can use these phrases even if they aren’t entirely true—consider them mild deceptions for the greater good. Pretexts are discussed further in the sidebar “Using a Pretext.”

Table 2.2—What to Say: Taking a Break For a usability study with a multiroom setup:

For on-site and remote sessions:

USING A PRETEXT

While you should avoid misleading the participant, there are times when the truth is going to be harmful. For example, if you notice that the participant does not have the characteristics you need and you want to consult with your stakeholders—who are in another room—about what to do, you should not express your concern about his qualifications directly to the participant. Being honest may hurt his feelings or make him nervous about participating. Instead, make an excuse to leave the room.

Technology can be blamed for many things and provides a convenient excuse if you need to leave the room or end the session early. Just make sure that you have something realistic to blame so that your reason for leaving isn’t obvious. For example, if you say, “The computer isn’t working correctly,” the participant may realize that you’re lying because he can see that it actually is working. But if you say, “Our prototype isn’t working the way it’s supposed to,” he does’t have any reason to know that it is just an excuse.

Examples of what you might say include:

If you don’t feel comfortable blaming the technology or it isn’t applicable—for example, in an interview where you earn’t using any equipment—you can use the study itself as a pretext:

2.2 Resolve any threats to physical safety

In the middle of a contextual inquiry, you hear a low rumbling sound. You take a moment to evaluate the situation, and quickly realize that you’re experiencing an earthquake. What do you do next? (For more about how to handle earthquakes and other natural disasters, see section 14.2.)

When a situation involves a physical threat, you need to respond to and address the threat as soon as you’ve taken a couple of seconds to understand what’s going on. Respond quickly and appropriately, even if that means that you’re unable to continue the session. The safety of you and the participant takes priority over your data collection needs!

Keep in mind that the participant may not know what to do, especially if he is in your facility. You must take responsibility for his safety as well as your own. For example, if a fire alarm goes off or your building is evacuated, bring the participant with you as you follow standard evacuation procedures. While the participant may end up seeing your observers, your top priority is to keep him from physical harm instead of maintaining the integrity of the session.

In the example of an earthquake, you should instruct the participant about where to go—for example, under a desk or in a doorway. If you’re moderating a usability study from an adjacent control room, and you think it’s safe to spend another couple of seconds using the microphone, tell the participant that you’re experiencing an earthquake and instruct him to go under the desk or in the doorway immediately.

If you’re in the participant’s space, follow his lead to exit the building safely or follow the appropriate safety protocol for the event that you’re experiencing.

Depending on the severity of the situation and safety of the environment, you may want to cancel or end the session rather than asking the participant to wait around until the situation is resolved. If you do this, be sure to give him his full compensation and thank him for the feedback provided. If the environment is safe, and the participant is not visibly shaken and willing to continue, offer to take a short break before resuming so you and the participant can both regain focus.

2.3 Verify that you’re not causing or magnifying the situation

During an interview, the participant starts making snide remarks after each question you ask. You’re not quite sure why he is responding that way. The issue does’t resolve itself, and taking a break does’t help. There is no physical danger to resolve but you’re worried about the participant’s attitude and its effect on the feedback he is providing. What, if anything, should you do?

Let’s step into some awkward territory for a minute, but we promise, it’s for your own good! Sometimes a situation is magnified, if not created, by something that you’re doing. Part of being a good moderator is being humble enough to constantly evaluate and reflect on how you’re interacting with participants. Before you jump to a conclusion about why the participant is behaving in a certain way, try to detect anything you may have done that led to this behavior.

Table 2.3 shows some examples of problematic moderator behavior that we’ve observed, the moderating styles (as identified by Table 1.1) that may encourage this behavior, and the effect it can have on the participant’s comfort level.

Table 2.3—Examples of Problematic Moderator Behavior and Its Effects

Moderator Behavior (Moderating Style) How the Participant May React
Over-the-top fake (Friendly Face) You’re trying to be very kind and respectful toward the participant while maintaining a friendly tone. However, you don’t realize that you come across as patronizing or disingenuous.
Buddy-buddy (Friendly Face) You want the participant to be comfortable, so you start acting like the two of you are old pals. You laugh together and find yourself making jokes. He tries to make you laugh and looks to you for affirmation, as with any natural friendly exchange. For an example of what this may look like, see Figure 2.1.
Obliviously abrasive (Down to Business) You maintain a serious, expectant tone as you ask questions. However, your tone comes across as overly stern. You also stare intently at the participant without looking away or blinking while he talks. For an example of what this may look like, see Figure 2.1.
Death by silence (Down to Business; By the Book) You’re trying to be as neutral as possible so you take a very minimalistic approach to moderating. You’re completely silent most of the time, using only brusque probes—for example, “And…?”—even though the participant is obviously nervous and uncomfortable with the silence.
Clock watcher (By the Book) class=“last-row”You’re trying to stay on schedule, and look at the clock or your watch at the most inopportune times. You ask a question from your study plan and, while the participant answers, look at the clock, look ahead in the study plan, or obviously pay no attention to the participant’s answer.

As discussed in Chapter 1, there is a delicate balance between the goals and method of your study, the type of participant you have, his current emotional state, and the appropriate moderating style to use. These last two elements in particular can change multiple times throughout the session. You should always carefully monitor how the participant responds to avoid making him uncomfortable. Again, part of the art of moderating is in evaluating your technique throughout the session and adapting it as necessary.

Figure 2.1—Examples of moderating behaviors—the Obliviously abrasive and the Buddy-buddy

Examples of moderating behaviors—the Obliviously abrasive and the Buddy-buddy

© 2013 Mark Ainscow, used with permission.

LEARNING TO ENJOY THE SILENCE

Most moderators are uncomfortable with silence during a study, especially new moderators! You may feel especially uncomfortable if you’re using a think-aloud protocol, since it may seem like the participant forgot your instruction to share what he is looking for or expects to see.

However, silence can be a valuable tool. Sometimes allowing a participant to quietly process and then answer the question or resume thinking aloud is more valuable than probing immediately for his thoughts whenever he becomes silent. For example, most people have difficulty thinking aloud while reading thick blocks of web content, so hold off on your questions while the participant is obviously reading. Similarly, not answering the participant’s question right away can lead the participant to answer it on his own. However, if he shifts toward you, or repeats the question while raising his voice, it’s time to say something.

It takes practice to become comfortable with this approach, but it pays off when the participant does’t feel rushed or harassed by your frequent reminders to talk about what he is thinking.

To find out if you’re creating or contributing to a problem:

  1. Check your body language:
  1. Replay what you’ve said during the session:
  1. Replay your paralanguage—how you said what you said:
  1. Evaluate your note-taking behavior:
  1. Check the physical setup of the room**:**

We’ll talk more about how to build awareness of your own behaviors while moderating in Chapter 16.

2.4 Check the participant’s comfort level

Once you’ve verified that the situation you’re seeing hasn’t been caused or exacerbated by your own behavior, check to see if the participant’s degree of comfort has declined to an unacceptable level.

Most user research starts out being somewhat uncomfortable for the participant. He may arrive without knowing exactly what he’ll be asked to do, or he may worry that he is being tested in some way. If you are coming in to his space, he’ll be unable to get away from the typical hassles of his environment, which may become magnified by the uncertainty of what he’ll be asked to do during the session.

Unless you’re doing very specific research that deliberately puts users in uncomfortable situations, you want to minimize, if not altogether eliminate, anything that compromises the comfort or safety of the participant. Part of your job as a moderator is helping the participant feel comfortable participating and providing his feedback, and ensuring that he remains comfortable—physically and psychologically—during the session.

PROTECTING PARTICIPANT RIGHTS

Most professional organizations whose members interact with nonmembers in a professional capacity, including the User Experience Professionals Association (UXPA), have a code of conduct or set of ethical guidelines to govern their members’ behavior. These guidelines are in place to ensure the ethical behavior of the professional—that is, you as the moderator—and protect the rights of the nonmember (participant).

As a user researcher, you need to keep your ethical obligations to the participant front and center throughout the session. Remember, when the participant comes to your session, he may have little to no information about what he’ll be doing and why he’ll be doing it. It’s your job to provide him with this information before he begins, and obtain his consent to participate.

This informed consent lets the participant know what he’ll be doing as part of the study, what he’ll receive for participating (compensation), and his rights—for example, to take a break or leave at any time. Even though you should have him sign a consent form before the session begins, review the key points verbally as part of your briefing at the beginning of a session.

We recommend reviewing the UXPA’s code of conduct, which outlines how to behave ethically as a user researcher. As a quick cheat sheet, here are the key things to remember when interacting with your participants:

Ensuring the participant’s physical comfort is usually fairly easy to accomplish. Let him know that he is welcome to take a break at any time and, if possible, offer him a beverage. If he seems uncomfortable in the research space—for example, because the chairs are at an awkward height or size for him or the air conditioning is running high—make whatever adjustments you can to keep the environment as welcoming and stress-free as possible.

The participant’s psychological comfort can be more challenging to maintain, especially when he is asked to interact with a product that has major usability issues or he is being interviewed about a sensitive or private situation. However, keep in mind that if a participant is struggling or growing frustrated with a usability study task, his comfort is not necessarily being threatened. In fact, that struggle often fulfills a common goal of user research, which is to identify problem areas. The participant’s struggle serves the purpose of making the product better for future users.

If, however, the participant berates himself for being stupid or unable to figure something out, or exhibits more than one of the signs of an uncomfortable participant (Table 2.4), you need to alleviate his discomfort. Remember that a participant who leaves a session feeling disgruntled or unhappy is an unhappy customer or potential customer. Given how easy it is to share negative reviews and experiences via social media, you want to avoid any potential damage to the credibility of your research or your organization.

If the participant seems uncomfortable, be careful not to assign any blame to him. Remember that he is likely in an unfamiliar environment (to him), or may be stressed to show a stranger (you) around his comfortable environment. He may be under a lot of stress unrelated to the session, such as childcare issues or tight work deadlines. Even if there is nothing external causing his discomfort, the research session itself can create some stress no matter what you do to avoid it.

Table 2.4—Signs a Participant May Be Uncomfortable

Watch Video 1 to see an example of a moderator interacting with a self-blaming participant during a usability study. The participant is very nervous about the session and is looking for affirmation.

Visit our Web site.

To help the participant feel more comfortable:

If the participant reaches a point where he asks to leave or end the session early, you’re obligated to wrap things up. Be sure to give him his agreed-upon compensation and thank him for his time and help. You can check on him afterwards—using whatever means you originally scheduled him for the session; if he was scheduled by a recruiter, ask the recruiter to follow up. If appropriate—for example, if he was distraught due to external circumstances—you can even offer to reschedule for a more convenient time.

SETTING CONTEXT FOR YOUR BEHAVIOR

The first few times a usability study participant hears you respond to his questions with questions (“What were you expecting?”), or when you do not respond at all, he may become unnerved (“I don’t know, that is why I’m asking you!”), or even sarcastic (“You’re not going to tell me, are you?”). Your lack of response may lead him to feel that he is being tested, as much as you say otherwise.

You can set the context for your behavior at the beginning of the session, and reinforce it along the way. In your presession instructions, tell the participant something like, “If you ask me questions, I may not really answer you or I may be vague. If I’m doing that, I’m not being unfriendly; I’m just trying to stay neutral….” Another helpful thing to say the first time he asks you a question is, “So I’ll actually turn that question around on you; what would you expect to happen there?” By doing this, the participant can start anticipating your moderating approach and will feel less like he is being evaded or tricked. Also remember that your paralanguage—how you say what you say, like your intonation, loudness, or tempo—can have an effect on the message communicated. There is a big difference between responding to a question with, “What do you think?,” “What do you think?,” and “What do you think?” The first may come across as inquisitive, whereas the second and third might come off as rude or patronizing.

If a participant keeps asking questions over the course of a session, we sometimes resort to subdued humor to make light of the situation. For example, we may kindly say, “I’m going to plead the fifth here—like I said, I’m just here to observe. Tell me what you’d expect it to do?”—works if you’re in the United States—or “I’m going to turn that around on you again! What do you think?” Again, if your delivery and paralanguage is kind and not patronizing, usually the participant will laugh along, stop expecting answers from you, and resume voicing expectations on his own.

2.5 Use careful language and tone to probe on the situation and begin to resolve it

Once you’ve established that you’re not causing or magnifying an issue, you’ll probably need to talk with the participant to try to understand more about, and hopefully resolve, the situation. One of the most important principles in moderating is to let the participant do most of the talking, but when you do speak—whether it’s to provide direction, probe for more detail, or give neutral reassurance—you need to choose your words, and your tone, carefully.

To choose the appropriate language, we suggest reviewing and memorizing some prefabricated responses that represent moderating best practices. Prefabricated responses are valuable for interacting with participants while maintaining neutrality and consistency. By memorizing at least some of these phrases and reviewing a list before your study, you save the cognitive effort of improvising appropriate responses on the spot. As we start reviewing specific situations in Part 2, you’ll be amazed at how flexible these short phrases can be.

The phrases we use most often include:

(For more examples of these phrases, see Appendix A.)

Since you need to be careful about providing positive feedback to participants, we’ve found the following phrases to be relatively neutral. When used judiciously, the phrases are capable of acknowledging to the participant that you’ve heard his feedback and are giving him a small amount of reassurance that the feedback was helpful:

Careful use of these phrases will address the majority of typical issues you see during your research. Just as important as what to say is what not to say, which we give examples of in the sidebar “What Not to Say.” Of course, you’ll encounter situations where these neutral phrases may not cut it, or if a participant does’t act as expected when given one of those phrases. In these cases, keep in mind these three guidelines:

Moderator: How would you do <task>? Participant: Oh, I would do <this>. Moderator: (pause, intonation rising) Oh. You would?

The rising intonation implies a questioning of the participant, and may cause him to feel judged or that he should rethink his answer. Instead, a graceful reaction would be to say “okay” with a neutral tone and move on, or ask him if there are other things he would try if the first attempt didn’t work. If he is interacting with a prototype, you could also let him attempt the task, ask how it compares to what he expected, and then ask if there are any other ways he might try to do the same thing.

Table 2.5—Examples of What You Should Not Say While Moderating

What Not to Say Why
“Let me give you a hint.” Reinforces that the participant needs help.
“Why would you want to go there?” or “What made you do it that way?” Implies that the place the participant wants to go, or did go, is not correct.
“Talk to me about… ” Depending on how it’s said, or how much it’s overused, can sound commanding or condescending.
“No, don’t <do/click> that.” Reinforces that the participant is doing something he shouldn’t.
“We/they did that because <special case could happen or special constraint limited us>.” Implies that you were part of the design team, and/or are defending the design choices.
“That is great feedback.” or“That is good feedback.” Implies that other feedback is not great.

2.6 Regain control to bring the session back on track

Hopefully, what you’ve done so far has jumpstarted the session. However, if the participant feels out of control or continues in a direction that is not useful to your research goal, it’s your job to steer him back. As the moderator, you’re always responsible for maintaining control of the session. This means accepting responsibility for changing the tone or providing whatever support is necessary to bring the session back on track.

Remember that you’re not the participant’s buddy. You didn’t set up this time with him for a gripe or therapy session. It’s your job to keep him from treating it that way. At the same time, some of the art of moderating includes redirecting a session and treating the participant’s feedback with grace and respect.

So how do you reestablish control of the session? For most one-on-one methods, a specific restatement or refocusing of a task or question is enough to get the participant back on track:

You can also bring the participant to a different task or question if you think he may be getting stuck or hung up on something in particular. (The pattern for redirecting the participant is more fully explained in section 3.3.)

If your first attempts to regain control don’t work, you may need to use stronger language to rein the participant back in:

Also, check that your body language projects an appropriate authority level. Maintain eye contact with the participant and angle your body toward him—this will help draw his attention and increase the odds that he’ll pay attention to you. However, never touch the participant, even as a way to get his attention or give him comfort. Remember that many people do not like being touched by strangers and the inherent power imbalance within a moderating session can make any touch completely inappropriate. (We talk more about how to use your body language in section 3.5.)

If the session is veering out of control through no fault of the participant—for example, technical problems—avoid sharing your frustration with him and do your best to keep that emotion out of your voice. The participant may feel like you’re directing that irritation at him.

Another reason you may need to regain control is when the participant does not meet the desired criteria and you must adapt accordingly. One approach to this situation is to transition the session into a different type of research. This works especially well if the participant is an important customer, or part of a user group, who you want to feel validated and listened to. For example, a usability study with a customer who does’t match your target user profile may be transitioned into an interview that will provide more information about his profile. Be sure to discuss these options with your stakeholders so they know what’s happening and can provide input on other types of feedback they might be interested in receiving. If, however, the participant will be unable to provide any kind of useful input—for example, if his role is entirely unrelated to your target—you may need to politely end the session. This technique is also referred to as shifting the focus, which we discuss more in section 3.8.

No matter what situations you encounter, the steps discussed in this section should help you figure out the best approach to address and resolve them. When in doubt, think back to what you would want done if your positions were reversed—how would you want your moderator to handle the situation? Chapter 3 details some of the specific interaction patterns that you can apply to help with this resolution.

SURVIVAL STORY: “I KNOW WHAT SHE NEEDS!”

Carolyn Snyder

I was running a paper prototype usability study at the client site. It was codiscovery. User #1 had already shown up. User #2 arrived with her angelic, sleeping two-month-old baby in tow. She was surprised when I reminded her that the session would take two hours. She was under the (sleep-deprived) impression it would just take a few minutes.

What I should have done was explained that we couldn’t accommodate infants, apologized for the misunderstanding, paid her, and sent her home. Instead, I asked the client if they were okay with a baby in the room. They were surprised, but didn’t object. (For the record, most of them were mothers themselves, and User #1 was female.)

Halfway through the session, the baby began crying. “I know what she needs!” the new mom proclaimed. Trying to be delicate, I asked, “Would you like privacy?” (For those keeping score at home, this was my second mistake: I should have taken her to another room.) “Nope, I’m good,” replied the mom, and put her baby to her breast right there in front of me, the other participant, and a handful of in-room observers. For the record, I am a proponent of public breastfeeding, but along with many other behaviors, it falls under the category of “Things I Would Never Do at the Office.”

I spent the rest of the session agonizing over how the client would respond. They chose not to make an issue of it—I have wonderful clients—but even now, nearly 20 years later, I still think about all the ways it could have ended badly.

It did cross my mind to send her home. But in the moment, it was surprisingly hard to choose that fork in the road. Being an eager young user experience consultant, I put a higher priority on getting feedback for the client than on the comfort of the participant, client, and even myself. I lacked the confidence to make a unilateral decision and take responsibility for it. I do still make mistakes, but I’ve learned to own them.

Discount for UXmatters Readers—Buy The Moderator’s Survival Guide: Handling Common, Tricky, and Sticky Situations in User Research and other Morgan Kaufmann titles on the Morgan Kaufmann site, using the discount code PBTY14, and save 25% off the retail price.

Topic: User Research

Join the Discussion

Asterisks () indicate required information. Your name: Email address: Remember personal info? Your comment: (You can use HTML tags for paragraphs, lists, and styles.)

Tags: user research ux research

Home