Running a beta pilot to successfully launch a (hardware) product
Listen to the episode
Now Streaming On
Explore the pivotal role of beta pilot testing in Canopy's journey and how it paved the way for redefining vehicle security.
Where would they park their trucks?
That was one of the many questions Anne Wilbers and the team at Canopy had to answer as they planned a beta pilot earlier this year for their new Canopy Pickup Cam product the company recently introduced aimed at preventing thefts from truck beds.
“It's really about providing peace of mind and having something that can protect your livelihood,” said Anne, a Senior Design Researcher at Canopy. “For example, for plumbers or carpenters, if their equipment gets stolen, they lose tons of money and they lose days of work because they can't work.”
In this Insights Unlocked episode, Anne shares her experiences UX testing the new Canopy Pickup Cam system ranging from the installation guides, hardware, app, website landing pages and more. They relied on insights from both an internal alpha pilot and a beta pilot with truck owners.
The Canopy Pickup Cam is a smart security camera that works for most trucks. Theft from trucks and other vehicles is a big deal. In fact, $7.4 billion was lost to auto theft and property stolen from vehicles in 2020, according to the U.S. Department of Justice. The company was spun out of Ford, where the initial research that went into the product was developed.
What is pilot testing
A beta pilot, also known as a beta test or beta phase, is a method of testing a product or service before it is released into the market.
Amrit Bhachu is a principal customer experience consultant for UserTesting. He helps companies, ranging from startups to those in Fortune 100, dial in their experience research initiatives.
Amrit ascribes to the fail fast, learn fast mindset. A pilot allows you to “get it wrong as quickly as possible.”
“When we're able to get it wrong as quickly as possible, that means that we can take those steps forward as quickly as possible as well,” Amrit said.
“What pilots allow us to do is to get through some of the more pernickety questions and help us adapt and iterate upon that really quickly,” he said. “So by time we do get in front of our actual users, our actual customers, we're in a better place and we're getting a better value of the time that we're spending with them.”
Why is pilot testing important?
The benefit of a pilot, whether an alpha or beta pilot, is that they help you better understand if your new product or service will be needed and wanted in the marketplace.
Pilot participants, called beta testers, use the actual product or service in real-world conditions to identify any potential problems, provide feedback and uncover any remaining bugs or glitches.
In Canopy’s case, their internal alpha pilot prepared them for the larger, real-world beta pilot.
“I really found the alpha pilot that we ran internally extremely helpful,” Wilbers said.
She said it helped identify who needed to be involved, assign responsibilities, and get buy-in from all the stakeholders.
How can pilot testing improve UX research?
This is sometimes referred to as testing the test. Your job as a UX researcher is to uncover honest and accurate insights about whatever it is your testing, whether a product, website flow or IRL service.
Pilot testing allows you to uncover where you may be asking biased questions or making inaccurate assumptions.
Anne said their internal alpha pilot helped her as a researcher do a dry run of her plan for the larger (and more expensive) beta pilot.
“There’s always things where you’re like, ‘Oh, actually, that’s not a good question. I should rephrase that.”
UserTesting’s researchers have learned that one of the key ingredients of a great test is performing a dry run of the test script (also known as a pilot test). We have one user go through the script, and then we watch the video as we normally would, keeping an eye out for any possible improvements to the script.
We check to make sure that:
- The user answered all of the questions that we needed answers to
- We didn’t accidentally use any confusing terms or jargon that would make a user stumble
- The user evaluated the right pages
If not, we adjust our test plans accordingly and try again. This way, we can ensure that our test plan is producing high-quality feedback from users and providing truly actionable insights for our clients.
Best practices for pilot testing
In the episode, Anne shares her experiences over the past year getting Canopy’s UX research program started, planning an internal alpha pilot and then a beta pilot with Detroit-area truck owners—a continent away from her offices in London.
She includes her lessons learned, such as the benefits of having an alpha test. And, she said, be ready to plan more than you’d originally would expect. For example, the team rehearsed before the beta pilot launch, so they could address details such as where participants would park their trucks, who would greet them, and was wifi available?
“I personally love a good diary study,” Anne said. “And I think that was definitely the best choice method wise over this three-week period, just because you can see the ups and downs and discover things like, ‘Oh, what happens after three days?’ Does it get better again after six days?”
She also described the beta as a “tremendous team effort” including folks from engineering, sales, marketing, product designers, and more.
“We all want to learn fast, right? And when things go wrong, you want to know this as soon as possible,” Anne said. “So we had dedicated Slack channels where we every morning went through the diary study videos that came in and were like, ‘Oh, this failed, or this went wrong.’”
They would share all this with relevant stakeholders, and also write a weekly report they shared with the whole company every Friday.
“I really like to share positive things as well because it's not all bad,” she said. “There's a lot of great feedback that you get.”
Related articles for running a pilot
Looking for additional help or information on pilot testing? Check out these related articles from UserTesting.
Episode links
- Anne Wilbers on LinkedIn
- Amrit Bhachu on LinkedIn
