Usability vs Beta Testing (The What & When)

If you work in software and product development (or are stumbling around a UX blog…) you’ve probably heard about usability testing and beta testing. They sound kind of useful and buzzwordy, but what do they actually mean? Are they even different from each other in any meaningful way? I’m asked this question quite often, so I wanted to define what each one is — what each one is not — in the life of a UX researcher at a SaaS software company.

Usability Testing

Usability testing is the process of watching an actual user use your product/screen/app to see if it’s usable.

What does that mean? It means that you may be sitting in a room with the developer, designer, product manager, and real life customer, tears silently running down your face because the user can’t figure out how to send an email with your sleek new interface. It can be painful, but a good type of pain. Watching a user not able to find the “Big Blue Button” at the top of the screen may make you stare at your colleagues in disbelief, but that’s because you know exactly where it is… because you put it there. And if enough people can’t find that “Big Blue Button”, you can’t blame them. You have to blame yourself. Or let’s just say “your design.” Yeah, your design probably needs to change.

Some say that usability testing is too contrived to be useful because it forces a user to concentrate on a task more than they typically would. But if a user doesn’t understand how to do a thing when they’re concentrating on it with you in the room, they sure as heck aren’t going to be able to understand it when they’re alone. Usability testing allows you to ask questions on the fly to really get into why someone is doing what they are doing.

When to usability test: During a design phase, gathering baseline data, or when you have a question you want to answer.

Usability testing is great when a team has a design that hasn’t yet been fully developed but would benefit from some exposure to real users. Testing with clickable mockups allows you to show someone a potential feature or design before any real (and costly) development work has been done.

It’s also great to simply sit down with users and ask them questions before the design phase has even begun. This kind of user-driven and user-informed research allows you to see how people are already doing things on their own, what workarounds they’ve developed in their daily lives, and whether or not it’s even a good idea to spend time thinking about a new feature. This isn’t formal usability testing in a lab with cameras and prototypes, but is more like just a simple conversation about what people are already doing.

Usability testing is also helpful if you:

• Have some hypotheses about how users do something in your product and want to watch them use it to confirm or disprove your theory
• Want to gather deep insights by asking pointed questions about a particular feature from a few targeted users
• Want to watch users use your product as a fly on the wall to see confirm if they are using the features as intended
• Want to test designs against each other
• Made changes to a design from a previous usability testing session and want to test if you went in the right direction (Were users able to complete more of the tasks you’re testing? Are they confident in what they’re doing? Do they understand what to do on a screen without additional explanation?)

Beta Testing

Beta testing puts the completed product/app/software into the hands of end users in order to see whether these beta users will actually use the new product or feature once it has been released while gathering feedback and bugs in the process.

Beta testing allows you to ask users questions after they have the new software, track usage and file bug reports. Often there are actual questions in mind you want to answer, like “Does this feature help you accomplish x? How are you using it to do that?” or “Is [feature x] something you have utilized yet in your account? If so, how did you set it up?”

If your users don’t have the answers to these questions, it’s probably a sign that they aren’t using the new feature. Give them time and follow up in a week. If you can track the percentage of people using a feature, keep an eye on usage over time. Is it changing or staying stagnant? If your beta-tolerant invested testers aren’t using your features, chances are that your regular customer base will use it even less.

When to beta test: When your product is built and you want to immediately put it in the hands of the end users to gather feedback.

At HubSpot, we have a few hundred customers who have opted in to beta test new features. Once a feature is built and ready to go into the hands of the beta group, we send an email letting them know that the feature is now live in their HubSpot account. We give them links to get there, explain how it works, why we built it,and include direct questions about we want to learn about the feature and how they can contact us. Often, they’ll reply directly to the email and it will go to a real person (me), but we’ll also give them a link to a thread about the topic in our forums for discussion. From there, we can track usage and solicit feedback.

Beta testing is helpful if you:

• Already built a feature and want to ship it to a subset of your users to gather feedback, questions and bugs from different sources and personas
• Have a specific question you want to answer about a live feature in order to make informed changes
• Want to have many people using it immediately
• Want to track usage and adoption rates in a subset of your whole customer base

Let’s sum it up.

Usability testing shows you if something is usable, and beta testing shows you if people will actually use it.

Here, I made us a chart.