Moderating a Usability Test

Moderating a usability test is a critical skill for all UXers. Whether you are a dedicated researcher or an interaction designer, the ability to gain feedback, with minimal bias, is critical to putting the 'user' in user experience. I find the practice of moderation to be fascinating and have had the opportunity to undertake literally thousands of user tests.

Watch a great practitioner and you'll hardly know they are working except that over time you'll notice their consistent ability to elicit feedback, overcome challenging people and situations, as well as their ability to get to the heart of a problem. Conversely, watch a beginner and you'll see how difficult the art can be to master. That said, with a little practice and a solid understanding of the basics anyone can quickly learn how to run a usability test and begin the journey to mastery. To help you on your way to effectively moderating a usability test, here are a few tips.

1. Start on the right note

The key to a successful session is engagement. From the minute you first interact with the participant it is crucial to set them at ease and make them feel comfortable. This rapport-building is about gaining the participant's trust as well as letting them know you are in charge. Mutual trust building helps participants feel comfortable that the session will be run in a manner where they are not going to be judged or made to feel uneasy. This comes down to a friendly but professional demeanour. Rapport can be built quickly, you need to be friendly and warm, be aware of your body language, ask some open questions about their day, or job and actively listen to their small talk and reflect that you are interested in what they have to say.

2. Make the process crystal clear

Setting boundaries with participants as well as providing a clear understanding of the process is crucial to making sure that participants know what to expect and what is expected of them. This can overcome some potential issues regarding the anxiety some participants feel about undertaking a 'test', the desire of participants to please the moderator (i.e. telling you what they think you want to hear) and their potential diversion into trying to solve the design issues during the session.

A good usability testing script or guide can be utilised to communicate key information to participants, including:

• That you want to understand how they would use the interface in their typical environment.

• That you will evaluate (and ultimately improve) the website by observing their interactions with the site and with this understanding, you will work out how it needs to be designed; make it clear that the participant doesn’t need to worry about re-designing the site during the session.

• That you are not testing them. A successful usability test is one where participants behave as normally as possible (or as closely as is realistic in an unnatural setting). Other tactics can be employed such as avoiding the term “task” and using “activity” instead. Also, avoid comments like “good” or “well done”. While it is unrealistic to totally eradicate the sense that they are being tested, it can be minimised.

• That you are independent of the design and that you won’t be offended by negative feedback. Independence is crucial to promote honest feedback (see Usability Testing: Does independence matter?) The reverse can be true, making out that you expect negative feedback can also adversely affect a session.

3. The masterful art of deflection

Some participants will seek out assistance from you as the moderator (a good introduction script should reduce the likelihood of this happening). When this happens the key is to deflect the question while maintaining the participant’s engagement. For example deflecting the question poorly such as, “My role is not to answer your question” can actually do harm by making the participant feel that their feedback is unimportant. A better approach is to use phrases such as “What do you think?” or “Let’s discuss that in a moment”. Maintaining a respectful tone of voice is the key to deflecting the question.

4. Avoid leading participants

Eliciting non-prompted and honest feedback is fundamental to the role of effective moderation. The way you phrase your questions to participants can lead to completely different responses and it is important that participants are not led to an answer. An example of a leading question would be “Did you find creating a password difficult?” The use of a non-leading question such as “How did you find creating a password?” will elicit a more true response. Watch this classic clip to understand a researcher's ability to influence people's feedback.

5. Get comfortable with silence

Silence is one of the most effective moderation techniques. As outlined above, questions can impact a participant’s response, especially during a task. This presents a challenge, as any stakeholder viewing the session will want you to ask why, “Why did the participant select that option?” “Why didn’t the participant sign up?” etc. Instead, a better choice is to encourage natural behaviour by deferring discussion to after the activity or to an appropriate pause.

Silence is difficult, and sometimes you want to encourage your participant to continue – using some minimal encouragers, such as “I see”, “tell me more about that”, “and then…”, are a nice supplement to silence if you feel your participant needs encouragement without impacting on their train of thought or their subsequent behaviours.

It can be argued that asking more probing questions is likely to reveal more insightful feedback and at times this can be of value but you need to ask yourself, will asking this question now have an adverse impact on behaviour? If so, does the insight gained outweigh changing their natural behaviour.

6. Listen

As someone much wiser than myself once said, “We have two ears and one mouth so that we can listen twice as much as we speak” and when moderating we should follow this advice, and then take it to the extreme. Active listening will make your participants feel comfortable, make them feel heard, and allow you to build trust, hopefully supporting them to participate in the test more fully. The best way to start active listening is to genuinely listen and show an interest in what the participant is saying (don’t tune out – remember you don’t have to be friends and it will be over in 60 minutes or so!). You can show an interest in their responses through your body language/posture, eye contact, use of minimal encouragers, asking open questions and use of attentive silence. Finally, active listening requires the moderator to get some feedback from the participant that their message is being understood in the way it was meant to be. This can be done by reflecting back to the participant what it is that they just said e.g. Participant: “I didn’t like setting up the password, it was hard” – Moderator: “So you are saying that the password set up was challenging?”. Apart from this showing the participant that you have actually listened, it allows you to check that you have understood what the participant has communicated to you, validating your conclusions. See Active Listening: Mastering the art of engaging conversation to learn more about active listening techniques and for some activities you can use to practice.

What now?

Mastering the art of moderation involves running lots of sessions and making lots of mistakes and can take years. That said, learning to gain useful insights from usability testing should be straightforward for anyone, regardless of your role. The most crucial elements are making it clear to the participant what is expected of them, starting on the right note and learning to listen.

As part of our mentoring and training with internal UX teams, we have found huge value in orienting or re-orientating them to these moderating basics. Understanding these tips has allowed team members to gain the confidence to start undertaking user research at all stages of the design process.