What is A/B Testing?

At some point or another you’ve probably heard about A/B testing and how great it is, but maybe you weren’t quite sure what it was or how it is done. 

An A/B test is a controlled experiment where you unleash two different versions of a design onto the world and let your users “tell” you which one is best. I like A/B tests because it helps me make user-centered design decisions; I can test which color of a button is best, which email subject line is best, the layout of a website, et al. A/B testing is powerful because it allows your users to make the decision for you and you to give them the best possible version with them in mind. 

You can test myriad different things from the subject line of an email, the images on a website, the content of an advertisement, to the color of a button. The possibilities are "almost" endless.

A How-to Guide to A/B Testing

Step 1: Set the goals - What do you hope to achieve from testing? What do you want to gain from the information you will gather? It is important that these goals are measurable so you can prove that your efforts are of value. 

Step 2: What to test & how to measure - Are you comparing two email subject lines or the placement of an image on a webpage? Decide what you are going to test and how you will know what success looks like. For an email subject line test, you may want to look at the number of opens as a metric or you could measure the click-through rate. 

Step 3: Create A/B versions - This step is pretty self-explanatory – once you’ve decided what you are going to test, create different assets.

Step 4: Run the test! - This is the best step because it is the point where you get to run the test and let the users do the work. 

Step 5: Analyze data & decide - Now you’ve run your test and gathered the data, now analyze it and see which version performed better!

Step 6: Implement changes (or do nothing) - You understand which version is the best, now you can implement any necessary changes.

Step 7: Test again! - Users change and it’s important to always be testing different components to better understand what is best for them.

There are a few different ways to conduct an A/B test, depending on what and how many variations you want to test: 

A/B Testing: This is where you test one version against another, typically one version is the original or current version. 

Multivariate Testing: Testing the original against many different versions; here you may have ABCD versions - all are variations of the same thing to see which would be best (think testing the color of a button).

*When conducting an A/B test, it is important to only change one thing between the versions so you can understand what is causing the difference in response. Example: If you want to test an email subject line, only change the subject line between the two emails. 

Why conduct A/B tests?

Low cost– it doesn’t require expensive tools or an expert, you only need two versions of what you want to test and a way to divide the users into two groups. 

Measures actual behavior- It is a test that causes no inconvenience to users – most won’t even notice something is being tested. Having users use what you are trying to test without knowing gives you the richest information about them. 

User-centered– I would argue this is the most important point.A/B testing helps you pick the option that is best for your user. By conducting A/B tests you are taking the guesswork out of design decisions and ensuring you make the right decision with your user in mind. 

Informed decisions– By using A/B testing, you eliminate the risk of “bad” decisions and ensure that you are making a decision informed by data that was accurately gathered so you know you’re making the right decision. 

Measures small differences– A/B testing allows you to test the smallest things, from the location of a button on your webpage to the color of a button. 

Creating the best version– You can constantly be checking and making sure your website, email, et al. is the best version it can be. 

Limitations of A/B testing 

A/B testing is great, but it is important for you to understand the constraints so you can make an informed decision on whether it is right or not for your goals

Doesn’t tell you why– A/B testing can tell you what works and what does not, but it cannot tell you why it does/does not work. To understand why you need to conduct deeper user research. 

Are you solving the right problem?– While A/B testing is a powerful tool that you can use to constantly test and improve, it cannot tell you whether or not you are solving the right problem. 

My experience with A/B testing 

I’ve used A/B testing in multiple different ways, from testing social media posting times to email subject lines, but my favorite A/B test I ran was from when I worked at Nintex.

During my internship, I wrote and designed a weekly blog newsletter that was sent to blog subscribers and prospective customers. I wanted to know how different layouts would compare, so I ran an A/B test in Marketo comparing the two email layouts. Following the tests, I analyzed the data using metrics such as click-through rate and where they clicked in the emails. Ultimately, I found that the original layout performed the best and we went forward with that design.

Here are the two layouts I tested:

Companies that do A/B testing well

Discovery Communications

Discovery Communications tested their video player. The outcome? A 6% rise in video engagement. 


comScore tested customer testimonials on their product pages and increased their conversions by 69%!

Electronic Arts

Through testing of promotional messaging, EA increased their digital downloads by 50% for their release of SimCity 5.

Extra Resources

The Harvard Business Review put out an article titled The Surprising Power of Online Experiments that goes into how to design A/B tests, test integrity, and how to analyze results. 

The Neilsen Norman Group article has some great information in A/B testing and how it compares to other types of testing in cost, benefits, risk, et al.