For most usability professionals lab-based research has been the only testing method they’d consider. However, with the growth of Web 2.0 and Web 3.0 technologies online, unmoderated user testing (also known as remote unmoderated or asynchronous user testing) is gaining traction and being used not only as a compliment to, but in many cases as an alternative to traditional lab-based research.
In 1997 when the internet was relatively new to the non-technical public market researchers started experimenting with online surveying to replace the costly and time consuming telephone and mail surveys. There was much resistance in the market research industry to online surveying, and it was often seen as a “cheap and dirty” way of conducting market research studies. Despite benefits such as cost, speed and geographical reach, to name a few, many market researchers dismissed online surveying and chose not to offer it as methodology to clients.
Today, there is no doubt online surveying is a common tool in the suite of products offered by market researchers. Clearly, they just needed time to understand the internet and online surveying to know how to get the most out of the new methodology.
Online user testing is in much the same place today as online surveying was in 1997; resisted by many and even dubbed a “voodoo measurement technique” by a very well respected user interface engineer. We think he’ll eat his words, just like Thomas Watson, Chairman of IBM did a while ago.
Those who have explored and experimented with online user testing are currently reaping the rewards. In a recent study by the Usability Professionals Association online user testing experienced an 18% increase in usage by UPA members since 2007, while traditional usability testing experienced a 9% decline over the same period.
Online Unmoderated vs Online Moderated User Testing
There’s online, moderated testing and then there’s online, UNmoderated testing. The key difference is that with online moderated testing the moderator uses a web-based service that allows users in faraway locations to participate in what is essentially, an online meeting. The software has the ability to share viewing and control of a web browser that permits the moderator to view the user’s mouse movements and web pages they visit, facilitating discussion similar to that of a traditional lab-based usability test. A good visualisation of the approach can be found here.
In contrast, online unmoderated testing is usually done ‘asynchronously’. That is, first the researcher designs and initiates a study; the participants perform the tasks in their natural context, at home (or the office) using their own computer; then, once all the participants have completed the tasks, the researcher gathers and analyses the data. In this approach there is no need for a moderator to be present during the testing.
Benefits of Online, Unmoderated Testing
There are a lot of great reasons to do some online, unmoderated testing on your website. Here are a few of the key benefits:
Get quantitative usability metrics
Perhaps the most important benefit of unmoderated testing is that it allows for the collection of quantitative usability metrics; in other words, statistics. Traditional lab-based testing often consists of between 5 to 8 participants; perhaps double that if you have a bigger research budget. With such a small sample size it would be wrong to attempt to calculate percentages to assist in reporting, such as the percentage of participants who completed a task successfully or the average time it took to complete a task. The larger sample sizes that are possible with online testing make the calculation of these and many other usability metrics entirely possible, and these metrics can deliver significantly greater insights into the usability of a website than ever before.
If you run some online testing in conjunction with lab-based testing you’ll get the chance to validate your lab findings to ensure they are a valid and true representation of your websites’ user experience.
Conduct benchmarking studies
Because online testing allows for the collection of quantitative usability metrics it becomes very easy and practical to measure how one website performs against competing websites or other versions of the same website. This can be particularly valuable if you are redesigning your website and have prepared wireframes of a new design. Pit them against each other to ensure the new design performs better on those key tasks your site visitors come for.
Test with ‘hard-to-get’ participants
If the target audience of your website is senior business executives, doctors, lawyers or brain surgeons then getting them into your lab for testing can be very difficult and very expensive if you’re offering incentives. But getting these people to do a 5 to 10 minutes online user test in the comfort of their own office is a much more realistic proposition.
Similarly, if you want to test your website for accessibility by doing testing with people who use assistive technologies (screen readers, electronic pointing devices, alternative keyboards, etc) you’ll never be able to replicate everyone’s unique set-up in your own labs. Here’s a great opportunity for online testing to step in and take over.
Test with an international audience
Similar to the above point, if your website has an international audience and you want to ensure it can account for any cultural differences that might exist, lab-based testing around the globe is guaranteed to blow your budget…if you had one big enough to begin with.
Online testing easily allows you to recruit participants anywhere in the world. And if you don’t have a database of your own there are a slew of market research panels that will provide participants that fit your specific criteria.
The ‘Test early, Test often’ principal becomes a reality
With all the different web-based tools around these days that can assist in user research the cost of doing usability testing is always going to be much cheaper than the lab-based alternative. And once you’ve run a few online studies you’ll be able to run them much faster than anything you can do in a lab.
It follows that if it’s faster and cheaper, you can now conduct more user testing studies more frequently with your currently budget. For years they’ve said, “Test early, test often”, but how many of us have honestly been able to do that? Now, it’s become a reality.
Below is an excellent example of a systematic and thorough usability testing process (thanks Grundyhome.com). There are eight stages of testing in total. If you use Loop11 for all the stages of task testing, Survey Monkey for the surveys, Optimal Sort for card sorting and Treejack for the IA testing you could do all of this for around $2,500.
For those who want to read more about online user testing there are two books I recommend reading:
- Beyond the Usability Lab: Conducting Large-Scale User Experience Studies, by William Albert, Thomas Tullis, and Donna Tedesco
- Remote Research: Real Users, Real Time, Real Research, by Nate Bolt & Tony Tulathimutte. Anticipated publication date: 2009.
- Introducing Loop11’s Tester Panel - March 24, 2020
- Our Response to the Coronavirus - March 17, 2020
- Release Notes: 5 Second & First Click Tests, Image Testing& More… - October 17, 2018
Give feedback about this article
Were sorry to hear about that, give us a chance to improve.
