Seven Things Remote Testing Can Do That Lab Testing Can’t.

The influx of remote usability testing tools has started to change the way usability testing is conducted. There are new techniques and different metrics to consider when conducting user research now.

This post is not aimed at polarizing people or making people pick remote testing over lab based testing, it has been written to differentiate some of the capabilities of the two methods.

So we put a list together of 7 things you can do with remote user testing that you can’t do with lab based testing. Firstly, it’s important to understand that when we talk about remote usability testing tools, we mean full service, unmoderated “quantitative tools”. To understand the difference, you can read Bill Albert’s article on remote testing tools.

Seven Things Remote Testing CAN DO That Lab Testing CAN’T:

1. 1. Test early, test often…affordably

Remote testing is much more cost effective than lab based testing, which means you can test at every stage of the development lifecycle, from wireframes to a live website. Or conduct benchmarking studies on a weekly or monthly basis to see how changes to your website affect usability. Lab-based testing can cost tens of thousands of dollars, which usually results in lab-based testing not being run very often by most companies. Remote testing tools usually cost just a few hundred dollars, making it easy to test on a much more frequent basis, thus creating the opportunity to run more regular studies.

1. 2. True Intent Studies

True intent studies focus around a user’s real aim when using a website. What is their goal on your website, how do they go about it, and did they achieve their goal? These are the main objectives. True intent studies can only be run online because you must intercept users as they visit your website so you can observe their natural behaviour.

1. 3. Get large numbers of “hard to get” users in their own environment

You can test hundreds of people simultaneously, in their natural environment. Whether you need a doctor or French speaker in China, online panels let you tap in to the thousands of unique demographics out there.

1. 4. Get more in-house people involved

Remote usability testing is a great way of introducing usability testing into a company using limited resources and budget. Typically, most companies only involve a few people in lab-based usability studies. Remote online testing enables other parties to reap the benefits and even get involved. Quantitative metrics will show your IT guys why changes are needed to your website, and the finance guys will have the stats they need to understand why money needs to be spent.

1. 5. Get statistically significant data

Lab-based usability testing is all about identification of usability issues and qualitative reasoning, not about quantification. Due to the small number of participants in lab testing (usually 5-10) compared to the hundreds or even thousands in remote testing, one simply can’t generate the quantitative reports that high level company decision makers may want to see. However, with remote user testing you can quash any in-house differences of opinion by getting the hard facts about your website’s user experience.

1. 6. Special technology

   Some interfaces just don’t make any sense to test outside their intended usage environment. Maybe you are running a user test on Facebook. The test involves uploading personal work with personal email contacts, pictures, videos and more. For this, you need the users to have their own email contacts, photos, videos and more. Having them bring their laptop or media to a lab isn’t going to do the study justice. Remote testing would allow them to use their own equipment at their own place…saving time, money and improving the quality of the results.

Or, let’s say you’re testing a recipe website that guides users step-by-step through preparing a meal; it wouldn’t make much sense to take people out of their kitchen, where they’re unable to perform the task of interest. When this is the case, remote research is usually the most practical solution, unless the users also lack the necessary equipment. We also call this the participant’s “technological ecosystem” because it implies that their devices and computing environment have an impact on how they interact.*

*The term “technological ecosystem” and the above example of a recipe website was originally written by Nate Bolt in his article “Pros and Cons of Remote Usability Testing”.

1. 7. Online accessibility testing

Now, let’s say you’re still testing that recipe website, except this time you need to test it for accessibility. Meaning you need to determine if there are any barriers that might prevent people with unique physical or mental needs from using the step-by-step recipe website in their own homes. It would be virtually impossible to replicate the technology and assistive devices of users with special needs, so the best way to do this is online.

One of the best summaries of lab testing compared to remote testing was given by Nate Bolt in his June, 2010 post “Pros and Cons of Remote Usability Testing”.

As Nate said:

“Both in-person and remote UX research share the same broad purpose: to understand how people interact and behave with the interface you’ve made. There’s no need to set up a false opposition between the two approaches—one isn’t inherently better than the other. Despite the versatility of remote research, there are lots of reasons you might want to conduct an in-person study instead, most of which have to do with timing, security, equipment, or the type of interaction you want to have with participants.”

Happy Testing!