Chartio + UserTesting

At Chartio, our mission is to make data accessible to everyone. So our core product, which lets people query and explore their business data, needs to be easy enough for anyone to use.

Unfortunately, traditional business intelligence tools are notoriously complex. In fact, it’s been estimated that only 1 in 10 people can successfully work with them. And these tools are critical to asking and answering important business questions. Imagine if only 10% of people within a company could use Google. That’s the fraction of employees today who can actually ask questions of their company’s data.

We want to get that number up to 100%, so that everyone at a company can make informed decisions themselves without relying on data analysts or engineers for help. So we set out to build a B.I. product that would be intuitive and user-friendly enough that anyone could use it to query their business data.

In our early years at Chartio, we built a product that was more usable than most B.I. solutions on the market. But we were still noticing that many customers could only use Chartio up to a certain point; it remained too complicated for the majority of everyday users. So, a couple of years ago, we set out to build a new version of Chartio with an entirely visual interface to lift that usability metric from 10% to 100%. And we turned to UserTesting to help us do it.

Our first step was to build a prototype of our new interface. Then, we relied heavily on UserTesting to see if users could successfully work with it.

We started by giving people some sample schema data specifically, music store transactions. Then, we gave them a series of questions to try to answer using Chartio: What band has sold the most songs? What genre has the average highest price? The key metric we were looking at was chart completion, could these users successfully use Chartio to create charts to answer the questions we had supplied them with?

We ran an unmoderated user test. Everyone got one shot. We care about that one-time experience, and tested it until we got it right.

With that first prototype, we quickly learned that we were on the wrong track. So we built another prototype, and another. With each iteration, we would gauge its usability with UserTesting. As the number of prototypes grew to dozens, and then hundreds, we saw that usability statistic slowly but steadily increase. Soon 2 out of 10 people could use Chartio. Then, 4 out of 10.

After about 500 prototypes, we found that 6 out of 10 people could run queries using Visual SQL our name for our entirely visual B.I. interface. We kept making changes, and tested again. By late 2019, we had reached a point where 8 out of 10 users could now query data themselves, generate charts, and build their own data dashboards. It’s still not 100%, but it was close enough that we felt confident launching the new interface for all users in March of 2020

Again, before Visual SQL, only 1 in 10 users could successfully run queries and build charts from their data. Now, 8 out of 10 can. This is thanks in large part to the hundreds of user tests we were able to run with UserTesting.

UserTesting was critical in not just helping us gather data about our prototypes, but to understand the why behind what was working and what wasn’t. Data can be used to answer a lot of questions, but not everything. Instead, you need to be able to quickly test assumptions. Instead of building 500 prototypes, we probably could have built and released 10 full versions for the public and we would have wasted lots of time on things that didn’t work. Iteration is key.

Also, when there were disagreements about what was working, we could settle them with UserTesting. That’s when it clicked. There is a problem! It wasn’t deniable anymore. So, UserTesting definitely made our decision-making process more efficient. It also helped our whole team build empathy, getting everyone on the same page around our customers’ pain points and problems.

It’s still early, but the feedback on our new product so far has been great. Visual SQL has helped differentiate us in the marketplace, making Chartio the most robust and easy-to-use visual interface to data available today. That’s clear from our consistent industry-leading rankings on G2, as well as anecdotal evidence from customers who have told us their non-technical teams are loving the new Chartio. Some companies have even told us the product is so intuitive, some of their employees are now using it to learn SQL. (This was obviously not our intention, but a happy surprise nonetheless!)

We’ll continue pursuing that elusive 10 out of 10 goal, but for now, UserTesting has helped us come a long way towards fulfilling our mission of empowering everyone to successfully work with data.