Last week I did a session for the web teams on usability testing. Unfortunately I only had one laptop and the room was too small to get people sat in groups of 2 or 3 to try out moderating a usability test first hand. So instead I gave an overview of user testing including:
I’m particularly interested in what user testing methods we can use to effectively benchmark user tasks on our website, before and after we make improvements. I’m also interested in what testing methods might be suitable given some of the challenges we face as a local authority web team, for example:
So I covered off three types of user testing…
Face to face testing
Face to face testing (more commonly known as usability testing or usability evaluation) usually involves a moderator asking a participant to carrying out tasks using a prototype or live website, then observing their behaviour. Although usability testing is often conducted in testing ‘labs’, it is possible to do face to face testing very cheaply, in any location. All you need is laptop or paper prototype, pen and paper. But it can be hard to ensure all tests are recorded consistently. There’s nothing worse than trying to moderate and frantically scribble notes at the same time!
You can record sessions, ideally using software like Morae or Silverbackapp to capture the participant’s mouse clicks along with a video of their facial expressions, an audio stream and metrics. In my experience letting a stakeholder or developer observe the testing, or sending them a link to a video clip of the session, can be a powerful communication tool. I’m hoping we can record our sessions and show clips to colleagues in different Council services to help them gain a better understanding of the problems customers face in finding information relevant to their goals on our website.
Remote testing
There are two approaches to remote testing – moderated and unmoderated. Personally I haven’t carried out any remote testing yet and have been a bit sceptical in the past because:
But because of the cost of travelling to our customers and hassle of booking suitable rooms (especially at short notice), extensive face to face user testing is just not feasible on many of our projects. Remote testing offers the opportunity to test with more participants, more cheaply, and it provides metrics. So I do feel it’s time to bite the bullet and give it a go.
Remote, moderated testing
Remote, moderated testing works in a similar way to a web video conference, but usually the moderator can see the participant’s screen and facilitates the session over the phone.
Quite often the participant will need to install some software at their end, which can make the process more complex to arrange and therefore potentially unsuitable for the type of participants (i.e. external customers) that we would be recruiting.
There are a number of different software tools that can be used. Some work in a similar way to face to face testing, enabling the moderator and observers to watch the participant’s mouse clicks and cursor movements on the screen.
Remote, unmoderated testing
During unmoderated tests, the participant is required to carry out tasks on a live website or HTML prototype. A web-based tool generates task instructions and sits above or below the website in a separate frame. There are quite a few different tools available. Liz Bacon posted a very useful comparison of remote testing tools on the IxDA site (which I was reminded to dig out again this week thanks to a colleague). The tools can capture valuable metrics to measure efficiency (time on task) and effectiveness of tasks (task completion rate and error rate) as well as requesting the participant completes a survey and/or subjective rating questions after each task.
As well as identifying and learning how to use some of these useful tools and carry out remote tests, we need an easy method to recruit participants. Hopefully by advertising on our website and in libraries, we’ll manage to get a few willing volunteers. I am hoping that remote testing will give us a much broader reach, but I don’t anticipate it will ever remove the need for face to face testing.
Over the next few weeks we’ll be carrying out remote and face to face user testing on our website, so I’ll be able to feedback on how it goes and what we learn.
In the meantime here are some good resources on remote testing:
There’s also a new book coming out on the subject of remote testing soon.
JPros |
LCons |
|
Face to face |
•Can ask follow up questions
•Facial expressions and reactions convey more to moderator
•Stakeholders can observe
|
•Unnaturalcontext, participants may feel uncomfortable
•Participants keen to please you
•Cost of room and facilities
|
Remote – moderated |
•Reach geographically dispersed participants
•Reduced costs of travel expenses, room booking etc.
|
•Needs practise to moderate and handle technology
•Harder to establish rapport
•Can’t see non-visual cues
•User may need special software
|
Remote – unmoderated |
•Reach geographically dispersed participants
•Context more natural to user
•Larger sample to measure time on task, task completion rate
|
•Needs careful preparation
•Can’t see how user behaves – do they make a cup of tea or answer the phone in the middle? Realistic, but skews data
|
This content is published under the Attribution-Noncommercial-Share Alike 3.0 Unported license.
Michele,
Nice post comparing the various methods of testing.
One of the best tools for remote, unmoderated testing i’ve come across is Loop11 (loop11.com). Wondering if you have seen it and what your thoughts are?
Frank
Nice article – great overview of usability approaches… and great to see that localgov is taking usability seriously! Usability is so vital, but is tough to ‘build in’ when resources are so limited.
I’ll happily do a bit of remote testing for you if you like 🙂
Cheers,
Jon.
Hi Michele—
Great article! I work for a user research firm specializing in these remote methods, and we’re currently co-authoring a book about this topic, which you can check out here: http://www.rosenfeldmedia.com/books/remote-research/
We’ve also just started a website about remote research topics, and we plan to build it out in the coming months with articles and resources for remote research practitioners: http://remoteusability.com/
Would you mind if we syndicated this article through remoteusability.com? We’ll give due credit, naturally.
Thanks!
About the author