Why I Stunk at A/B Testing

I used to think it was just me, and it was really embarrassing.

As a marketer, I was supposed to be great at this. I was supposed to be able to run an A/B test, kick my feet up, and marvel at the success of the B-version.

Yet, it rarely worked out that way. No matter how confident I was in my newest test, it would only outperform the A-version of the site about 10% of the time. I was tragically worse than a coin flip.

When I finally started to reach out to friends for help, I realized that it wasn’t just me. Most of the people I talked to (and the blogs I read) made me realize that us marketers only create winning B-versions about 10-15% of the time.

But...why?

Why aren't our A/B tests moving the needle?

We’re a really smart group with knowledge of our companies, marketing strategy, psychology, and conversion rate optimization.

Why can’t we consistently produce new versions of our sites that really move the needle? And, how do we get better?

In an attempt to answer these questions, I went on the hunt for companies that had cracked the code. I knew that most people wouldn’t be willing to share their secrets, but I was hopeful that a few would. So I started to Google like a mad man, and read every A/B testing case study that I could get my hands on.

At first glance, most of the case studies I came across mentioned things like:

• Make your landing page longer
• Include pictures of faces
• Change the button to green
• Add a pricing chart
• Add a video

But, these solutions weren’t satisfying. I had tried a lot of these things before (and knew a lot of other people that had) and never received the same exciting results. So, I dug deeper, and I started to notice a trend that was….counterintuitive.

The secret to running an A/B test

I noticed that the specific changes companies were making to their sites weren’t important. It wasn’t important that the B-version was longer, or that it was a different color, or that it had a video. What mattered was how the companies arrived at those decisions. The secret to running a winning A/B test was all in the process. And, at the core of that process was a very simple idea: listen to your customers and prospects.

The companies that are the most successful at A/B tests start with these 4 actions:

• Listen to customers
• Listen to prospects (people who are deciding whether or not to purchase)
• Listen to people that have decided NOT to become customers
• Become your own customer

And then, they use the feedback from these actions to drive their A/B tests.

Why it works

This process works because, as marketers, it’s impossible for us to see our websites the same way our visitors do. We have the curse of knowledge. After looking at our sites each and every day, we can no longer see them with fresh eyes.

To create a B-version that moves the needle, we need to understand what’s going on inside the heads of our web visitors. Once we know what’s stopping them from converting, we’ll be able to fix it. Our B-version won’t be a guess, or a gut feel, or a copy of what someone else tried; it will be a response to feedback that we received from visitors, and it will be far more likely to succeed.

I don't stink anymore!

So, yes, I used to stink at A/B testing. But, now that I listen to customers and prospects before deciding what to test I've gotten a whole lot better. My B-version doesn't win all of the time (where's the fun in that?) but it's far more likely to succeed, and far more likely to make a big impact on the bottom line.

To learn more about how to decide what to test, check out our post about The Best A/B Test Idea You Haven't Thought Of.