Stop Making Excuses, Start AB Testing

If you are reading this, chances are you are not new to the idea of split testing, but just in case, here’s a quick and dirty definition:

A/B testing or split testing compares the effectiveness of two versions of a web page, marketing email, or the like, in order to discover which has better response rate or better sales conversion rate.

Source: Wikipedia

In my opinion, split testing should be the most frequently used tool in your CRO toolbox. Split testing alone is one of the greatest sources of ‘Ah Ha!’ moments and insights into exactly how your customers think about and perceive your brand. It has a lot to do with the way you optimize design to generate conversion optimization.

For some of my specific thoughts on conversion rate optimization, please check out my recent CRO interview with Ben Beck. Also, stay tuned for a special offer at the end of this post.

So What’s the Big Deal?

Simply put, if you are not actively and rigorously testing the stages of your conversion funnel, you are leaving money on the table.

The fact of the matter is, guessing is not testing.

While I am a firm believer that you should test vertical markets, testing for conversion is a different monster all together. If your website (or business) relies on conversions, at the end of the day, you are only as successful as your conversion rate.

What if a single test campaign could yield a 58% increase in conversion rate? What if that was only scratching the surface? I’m beating a dead horse at this point, but unless you are testing your hypotheses, that’s all they will ever be.

Stop guessing, stop making excuses, JFDI.

What’s With the Attitude?

Honestly, I just hate excuses.

What has become an almost weekly frustration for me, is meeting website owners, talking with them about their business, and asking them questions they don’t know the answers to.

Simple questions, like:

• How many steps are in your conversion funnel?
• What language resonates most with your audience?
• Which layouts have you found work best with your content?

More so, the reason they don’t have the answers is because they are not doing anything proactive to go out and get them.

This is exactly what A/B testing provides you with; the opportunity to get answers to your questions, for example:

• Want to know if a different color button/text will increase conversions? Test them.
• Wonder if you are not providing enough social proof? Test more.
• Is your workflow confusing your visitors? Test it.
• Is your headline not compelling enough? Test another one.

Worse still, are the ridiculous excuses website owners tell me in attempts to justify why they are not testing;

• They don’t have the budget
• They don’t have the technical expertise
• They don’t understand the value
• It’s just not a priority right now

Let’s Talk Through These

When you say “I don’t have the budget,” I hear “I haven’t bothered to look.” Google has a free split testing tool.

When you say “I don’t have the technical resources,” I hear “I haven’t researched any solutions.” There are insanely easy to use A/B testing tools.

When you say “I don’t understand the value,” I hear “I don’t understand what you’re talking about.” Companies have seen 100%+ increases from very simple tests.

When you say “It’s not a priority right now,” I hear “I think I’m a Guru and I enjoy hoping for success.” A bit harsh? Yes, but seriously… if this sounds like you, then you are probably one of the people that needs to ask yourself if you deserve a first page ranking.

AB Testing is Not Rocket Science

Again, I realize this may seem harsh to anyone who has not gotten started with testing, or is putting off getting started, so let me clarify:

• Designing tests can be very easy; start with your most obvious questions, and focus on variations that are simple.
• Split testing is very different from multivariate testing, just pick one element, and create an alternate version.
• Testing is most effective when lifts can be properly attributed; the benefit of split testing is that it focuses on one element at a time.

Where it Gets Technical

When it comes to testing, what seems to scare most people is actually not that complicated, but tends to involve math.

If you are not into math (which puts you in the vast majority) then phrases like confidence interval, statistical significance, and population size can immediately make your mouth dry and your vision blurry.

Let’s look at some of these a bit closer in an attempt to reduce the fear factor and shed some light on their importance;

• Confidence interval is used to communicate the reliability of an estimate. Just like with grades on a report card, a 90% is good, but a 95% is better; a 95% confidence interval means that the outcome of your test will hold up in 95% of scenarios.
• Statistical significance is a measure of whether or not your sample size (or population) is large enough to allow for assumptions to be made, specifically, that the outcome is based on a pattern rather than just chance.

The reason these measures are so important is that they are literally the measurements that impact the outcome of your tests. Your confidence interval will measure the likelihood of your testing pattern, and your population size will ensure that your sample is large enough to be statistically significant.

Translation: your test is effective and the outcome is accurate.

This is All Well and Good, But How Do I Get Started?

1. Choose a Testing Tool

This is often times the most daunting task for many folks as they look to embark on a split testing campaign. Your choice will vary on a lot of things, but more times than not, I’ve found it comes down to comfort with code and data.

If you are a Google Analytics™ wizard, and are comfortable with creating advanced segments and events, I might suggest Google Content Experiments.

However, if you are looking for an extremely simple, robust, and user friendly testing platform, I have to recommend Visual Website Optimizer.

I have been using VWO since the end of 2009, and in my experience, nothing is easier for designing, implementing, and managing split tests – this is why great CRO companies like Unbounce integrate their technology.

Additionally, if you are using wordpress, they have a very easy to use, free plugin.

2. Design Your Split Test

I find test design to be the most fun part of testing, well, other than seeing huge increases in conversion rates. But for many people knowing what to test is the most frustrating part.

I consistently get inspiration for new tests from what I like to call, reference cases. These are examples of other sites within your niche, or even more generally just within your space, that are commercially successful.

By commercially successful, I mean the big players – that are effectively closing in on their target conversions and creating revenue. It’s more than likely that these website have already invested a lot of time, energy, and money honing in on what works – so why reinvent the wheel?

I have found that using reference cases has allowed me to realize some rather large gains in short periods of time.

Analysis is simple; spend roughly an hour a day analyzing a collection of screenshots and looking for elements that your site lacks, things they are doing differently. You will eventually notice that patterns will begin to emerge, things like colors, functionality, and language. For more information on reference cases check out my post on Kaiser the Sage; moving from SEO to CRO.

3. Segment Your Population & Data Collection

You have some basic decisions to make before you deploy your test.

These have to do with who is going to see the test pages, how much traffic you want to test, and what you will measure as the conversion goals.

Some of the powerful features of VWO are:

• Geo Behavioral Targeting – Show different content to visitors based on the countries, cities, and even regions they are in.
• Heatmaps and Clickmaps – See where your visitors are hovering the longest and clicking the most. This is the same insight that provided Derek Halpern with the data to increase his conversion rate on Social Triggers.
• Split Testing URL’s – Split traffic between 2 variations of a landing page.

4. Specify Your Goals & Conversion Points

What good is testing if you do not have a clear criteria for success?

Before you deploy your first test, make sure you have a clear idea of what you want the outcome to be.

If you know that you want to increase sign-ups by 20% going in, this will change the way you approach the test and also your takeaways after it is over.

Tip: Don’t just track your main conversion. You have the opportunity to track any and all potential conversion points during your tests, and all data – is good data. VWO gives you the opportunity to track multiple goals beyond your main conversion point, including downloads, clicks, time on page, and more – see my example below:

5. Create a Testing Schedule

Tests are easy to lose track of, especially if you get addicted (like me) and start testing everything.

It’s important to create a testing schedule before you get started. It doesn’t need to be anything crazy, try to go out a month into the future if possible. If you simply can’t come up with that many tests, or don’t have enough traffic, shoot for a minimum of 2 weeks.

Use your baseline data to figure out approximately how long tests should take to complete. Also, check out this handy split test duration calculator.

On the flip-side, if you have a lot of testing opportunities and sufficient traffic, you will benefit from VWO’s ability to run simultaneous tests on the same pages. Not to mention real-time reporting and test intelligence can be invaluable for higher traffic sites.

What’s more is they have a pretty handy test scheduler that allows you to schedule tests to start and stop at future dates, pause after a certain number of visitors, or automatically re-occur every day, week, month, etc.

Let the Data Be Your Guide

So hopefully this post has inspired you to get started with split testing (or if you are already testing – to take it more seriously and start doing more of it).

One word to wise; make sure you to take action with your data once you have. The greatest downfall of testing is not letting your results inform your changes and shift your overall strategy.

Split testing gives us incredible visibility into the psyche of our visitors, and allows you to fine tune your website’s experience to meet the needs of the majority of your audience.

As always, please share your thoughts in the comments, if you enjoyed this post share it on twitter, and never hesitate to hit me up with any questions, critiques, or feedback.

Thanks for reading.